go
/
build
зеркало из https://github.com/golang/build.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
build/dashboard/builders.go

2227 строки
74 KiB
Go
Исходник Обычный вид История

dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
// Copyright 2015 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package dashboard contains shared configuration and logic used by various
// pieces of the Go continuous build system.
package dashboard
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
import (
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"fmt"
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
"os"
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
"sort"
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
"strconv"
"strings"
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
"time"
all: consolidate configuration for coordinator and gomote buildenv.Environment type defines configuration options: - Coordinator uses the GCE project name to lookup config. A custom config name can be provided at runtime to override. - The conventional prod and stage project names ('symbolic-datum-552' and 'go-dashboard-dev') map to prod and staging configuration structs. - Production and staging status is explicitly defined in configuration. - GCS bucket names for buildlet, logs, and snapshots are configurable. Change-Id: I7e6d7874eb0bdfe35dbdd5fcf6212ab50d576b88 Reviewed-on: https://go-review.googlesource.com/19502 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-15 02:59:59 +03:00
"golang.org/x/build/buildenv"
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
"golang.org/x/build/maintner/maintnerd/maintapi/version"
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
"golang.org/x/build/types"
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
)
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
// Builders are the different build configurations.
// The keys are like "darwin-amd64" or "linux-386-387".
// This map should not be modified by other packages.
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
// Initialization happens below, via calls to addBuilder.
all: change dashboard.BuildConfig to be a pointer type, like HostConfig BuildConfig grew way too large and has too many slices & such to be a value type. Make it a pointer type, which matches HostConfig. Change-Id: Ie625bece9d6d8c1ec6cff26e77416bd9b1f256d8 Reviewed-on: https://go-review.googlesource.com/c/145077 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 20:52:25 +03:00
var Builders = map[string]*BuildConfig{}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// Hosts contains the names and configs of all the types of
// buildlets. They can be VMs, containers, or dedicated machines.
var Hosts = map[string]*HostConfig{
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
"host-linux-jessie": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Debian Jessie, our standard Linux container image.",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
ContainerImage: "linux-x86-jessie:latest",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
"host-linux-stretch": &HostConfig{
Notes: "Debian Stretch",
ContainerImage: "linux-x86-stretch:latest",
dashboard: increase timeout scale for longtest builders, add more CPU Updates golang/go#25886 Change-Id: I5168e291ab77cbd3843bdc39e319a68dfa65aedd Reviewed-on: https://go-review.googlesource.com/c/build/+/167638 Reviewed-by: Russ Cox <rsc@golang.org>
2019-03-14 19:06:57 +03:00
machineType: "n1-standard-4", // 4 vCPUs, 15 GB mem
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
"host-linux-stretch-morecpu": &HostConfig{
Notes: "Debian Stretch, but on n1-highcpu-8",
ContainerImage: "linux-x86-stretch:latest",
machineType: "n1-highcpu-8", // 16 vCPUs, 14.4 GB mem
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "root",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
"host-linux-stretch-vmx": &HostConfig{
Notes: "Debian Stretch w/ Nested Virtualization (VMX CPU bit) enabled, for testing",
ContainerImage: "linux-x86-stretch:latest",
NestedVirt: true,
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
cmd/coordinator: cross-compile ARM build on Kubernetes Saves 4.5 minutes or so by using fast x86 machines to build the ARM build instead of running make.bash on Scaleway ARM machines. We still run the tests on ARM, and have a separate builder only running make.bash on ARM (see prior golang.org/cl/29670) Fixes golang/go#17105 Updates golang/go#17104 Change-Id: I1cb7b0e5b1cc8b644195f262328884ed3aff120a Reviewed-on: https://go-review.googlesource.com/29677 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-23 23:44:43 +03:00
"host-linux-armhf-cross": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Debian Jessie with armhf cross-compiler, built from env/crosscompile/linux-armhf-jessie",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-armhf-jessie:latest",
cmd/coordinator: cross-compile ARM build on Kubernetes Saves 4.5 minutes or so by using fast x86 machines to build the ARM build instead of running make.bash on Scaleway ARM machines. We still run the tests on ARM, and have a separate builder only running make.bash on ARM (see prior golang.org/cl/29670) Fixes golang/go#17105 Updates golang/go#17104 Change-Id: I1cb7b0e5b1cc8b644195f262328884ed3aff120a Reviewed-on: https://go-review.googlesource.com/29677 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-23 23:44:43 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
cmd/coordinator, dashboard: cross compile linux-arm-arm5spacemonkey Change-Id: I3a228627ec8236d0d10bb9a1a61443c81493145d Reviewed-on: https://go-review.googlesource.com/38460 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-23 18:46:39 +03:00
"host-linux-armel-cross": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Debian Jessie with armel cross-compiler, from env/crosscompile/linux-armel-stretch",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-armel-stretch:latest",
cmd/coordinator, dashboard: cross compile linux-arm-arm5spacemonkey Change-Id: I3a228627ec8236d0d10bb9a1a61443c81493145d Reviewed-on: https://go-review.googlesource.com/38460 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-23 18:46:39 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
"host-linux-amd64-localdev": &HostConfig{
IsReverse: true,
ExpectNum: 0,
Notes: "for localhost development of buildlets/gomote/coordinator only",
SSHUsername: os.Getenv("USER"),
},
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
"host-nacl-arm-davecheney": &HostConfig{
IsReverse: true,
ExpectNum: 1,
Notes: "Raspberry Pi 3",
OwnerGithub: "davecheney",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-nacl-kube": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Container with Native Client binaries.",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-nacl:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
"host-js-wasm": &HostConfig{
Notes: "Container with node.js for testing js/wasm.",
ContainerImage: "js-wasm:latest",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-s390x-cross-kube": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Container with s390x cross-compiler.",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-s390x-stretch:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
env: add alpine linux builder Adds flag to upload to build binary statically. Updates golang/go#17891 Change-Id: If19b2e2cf9a84fa2b4cc6fdf55d8add5b54abff7 Reviewed-on: https://go-review.googlesource.com/33890 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-02 22:14:12 +03:00
"host-linux-x86-alpine": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Alpine container",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-alpine:latest",
env: add alpine linux builder Adds flag to upload to build binary statically. Updates golang/go#17891 Change-Id: If19b2e2cf9a84fa2b4cc6fdf55d8add5b54abff7 Reviewed-on: https://go-review.googlesource.com/33890 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-02 22:14:12 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64-static",
env: []string{"GOROOT_BOOTSTRAP=/usr/lib/go"},
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-linux-clang": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Container with clang.",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-clang:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
"host-linux-sid": &HostConfig{
env/linux-x86-sid: convert from GCE VM to Kubernetes And update it. What we used to call "sid" was probably ancient. Change-Id: I76c1aad60eb655513119eaad0550ba5e0df3e9c5 Reviewed-on: https://go-review.googlesource.com/40399 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 04:54:09 +03:00
Notes: "Debian sid, updated occasionally.",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-sid:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
dashboard: builders.go change for new scaleway docker config Working towards golang/go#20397 Change-Id: I0861f0a265498840cd7e852504cd5b9e53893036 Reviewed-on: https://go-review.googlesource.com/43813 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-22 23:17:48 +03:00
"host-linux-arm-scaleway": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
IsReverse: true,
HermeticReverse: true,
ExpectNum: 50,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
ReverseAliases: []string{"linux-arm", "linux-arm-arm5"},
SSHUsername: "root",
dashboard: builders.go change for new scaleway docker config Working towards golang/go#20397 Change-Id: I0861f0a265498840cd7e852504cd5b9e53893036 Reviewed-on: https://go-review.googlesource.com/43813 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-22 23:17:48 +03:00
},
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
"host-linux-arm5spacemonkey": &HostConfig{
IsReverse: true,
ExpectNum: 3,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
ReverseAliases: []string{"linux-arm-arm5spacemonkey"},
dashboard: remove fake ARM5 builders The spacemonkey actually-ARM5 builders have been working and are better. The arm5 builders being deleted here were Scaleway C1 servers (which have better-than-ARM5 hardware) but running with GOARM=5-compiled binaries. That test a bit, but not enough. Fixes golang/go#19662 Fixes golang/go#19848 (looks like this is only missing owner annotation) Change-Id: I1d3b21b0139d4d4ccf2c0baebd4f2f01b4024861 Reviewed-on: https://go-review.googlesource.com/41618 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-25 02:07:45 +03:00
OwnerGithub: "zeebo",
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
},
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
"host-openbsd-amd64-60": &HostConfig{
VMImage: "openbsd-amd64-60",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-60.tar.gz",
Notes: "OpenBSD 6.0; GCE VM is built from script in build/env/openbsd-amd64",
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
},
dashboard: switch to OpenBSD 6.0 completely, remove OpenBSD 5.8 Updates golang/go#16951 Change-Id: I7b92363671e0ba4c6ab80cef60e7c8600687f440 Reviewed-on: https://go-review.googlesource.com/34384 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-12-15 01:41:25 +03:00
"host-openbsd-386-60": &HostConfig{
VMImage: "openbsd-386-60",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386",
dashboard: switch to OpenBSD 6.0 completely, remove OpenBSD 5.8 Updates golang/go#16951 Change-Id: I7b92363671e0ba4c6ab80cef60e7c8600687f440 Reviewed-on: https://go-review.googlesource.com/34384 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-12-15 01:41:25 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-60.tar.gz",
Notes: "OpenBSD 6.0; GCE VM is built from script in build/env/openbsd-386",
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
"host-openbsd-amd64-62": &HostConfig{
VMImage: "openbsd-amd64-62",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64",
dashboard: set openbsd builders to use a Go 1.12 bootstrap toolchain Updates golang/go#30633 Change-Id: I593a66e18f7546fdeaf7513e2a16bdf82a5fa5a8 Reviewed-on: https://go-review.googlesource.com/c/build/+/165747 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-06 22:45:35 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-go1_12.tar.gz",
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
Notes: "OpenBSD 6.2; GCE VM is built from script in build/env/openbsd-amd64",
SSHUsername: "gopher",
},
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
"host-openbsd-386-62": &HostConfig{
VMImage: "openbsd-386-62-a",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386",
dashboard: set openbsd builders to use a Go 1.12 bootstrap toolchain Updates golang/go#30633 Change-Id: I593a66e18f7546fdeaf7513e2a16bdf82a5fa5a8 Reviewed-on: https://go-review.googlesource.com/c/build/+/165747 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-06 22:45:35 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-go1_12.tar.gz",
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
Notes: "OpenBSD 6.2; GCE VM is built from script in build/env/openbsd-386",
SSHUsername: "gopher",
},
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
"host-openbsd-amd64-64": &HostConfig{
env/openbsd-amd64: force tsc time source, double the build speed Fixes golang/go#29223 Change-Id: I43d1fc1eb6455c063d3842c330cb7d57295d7482 Reviewed-on: https://go-review.googlesource.com/c/160319 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-30 08:01:54 +03:00
VMImage: "openbsd-amd64-64-190129a",
buildlet, dashboard: add min CPU platform knob, require Skylake for OpenBSD I thought this would be enough for OpenBSD to select the TSC on its own without being forced to (as in CL 160319), but apparently it is not: https://github.com/golang/go/issues/29223#issuecomment-459035482 So it seems like we want both this CL and CL 160319. Updates golang/go#29223 Change-Id: I0a092d62881d8dcce0ef1129d8d32d8f4025b6ac Reviewed-on: https://go-review.googlesource.com/c/160457 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-30 20:13:07 +03:00
MinCPUPlatform: "Intel Skylake", // for better TSC? Maybe? see Issue 29223. builds faster at least.
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64-64",
dashboard: set openbsd builders to use a Go 1.12 bootstrap toolchain Updates golang/go#30633 Change-Id: I593a66e18f7546fdeaf7513e2a16bdf82a5fa5a8 Reviewed-on: https://go-review.googlesource.com/c/build/+/165747 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-06 22:45:35 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-go1_12.tar.gz",
env/openbsd-amd64: enable SMT on OpenBSD 6.4 Disable Spectre/Meltdown mitigations and enable simultaneous multithreading (SMT) on the OpenBSD 6.4 builder image. This was done by setting hw.smt=1 in /etc/sysctl.conf when building the image. I considered an alternative approach of doing this at runtime via a cmd/buildlet special case for OpenBSD. It would've likely been faster to implement if I had the idea at the beginning. However, by the time I saw it, I had already started to build a new OpenBSD 6.4 image with the /etc/sysctl.conf file, then tested it via debugnewvm. At this point, there's no longer a time saving advantage, so I decided to prefer the v2 image because it keeps OpenBSD-specific configuration more contained in env/openbsd-* directory, rather than spreading it out between there and the cmd/buildlet runtime code. It seems to be a slightly better option. The times to do a full build, as measured via cmd/debugnewvm, were: 6.2 = 19m25.218s 6.4 = 28m49.565s 6.4 with hw.smt=1 = 22m55.909s That should translate to faster trybot runs, which is important for open source project health. The 386 environment doesn't need to be updated because it doesn't support hw.smt: sysctl: hw.smt: value is not available Attempting to set it anyway should be harmless and okay to do, in case it happens to get supported in the future. We're only using amd64 environment for trybots, so it's okay for the purposes of golang/go#28403. Reference: https://man.openbsd.org/sysctl.2#HW_SMT_2 Fixes golang/go#28403 Change-Id: I7bc4cbf83ccbdb3aa9dd19eeabd88feb1c425811 Reviewed-on: https://go-review.googlesource.com/c/145022 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 20:05:59 +03:00
Notes: "OpenBSD 6.4 with hw.smt=1; GCE VM is built from script in build/env/openbsd-amd64",
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
SSHUsername: "gopher",
},
"host-openbsd-386-64": &HostConfig{
dashboard: update OpenBSD 6.4 builders to final release Previously, they were running a 6.4 snapshot release from October 4, 2018, added in CL 139857. The final release of OpenBSD 6.4 has been released on October 18, 2018, and can be used now. See https://www.openbsd.org/64.html. Fixup unbound variable in env/openbsd-amd64/make.bash: $ ./make.bash ./make.bash: line 57: PKG_ADD_OPTIONS: unbound variable The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: I0e3b1ed5aa9b2e624d9afbe46a8a344f88ddcc56 Reviewed-on: https://go-review.googlesource.com/c/143458 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-19 22:50:28 +03:00
VMImage: "openbsd-386-64",
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386-64",
dashboard: set openbsd builders to use a Go 1.12 bootstrap toolchain Updates golang/go#30633 Change-Id: I593a66e18f7546fdeaf7513e2a16bdf82a5fa5a8 Reviewed-on: https://go-review.googlesource.com/c/build/+/165747 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-06 22:45:35 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-go1_12.tar.gz",
dashboard: update OpenBSD 6.4 builders to final release Previously, they were running a 6.4 snapshot release from October 4, 2018, added in CL 139857. The final release of OpenBSD 6.4 has been released on October 18, 2018, and can be used now. See https://www.openbsd.org/64.html. Fixup unbound variable in env/openbsd-amd64/make.bash: $ ./make.bash ./make.bash: line 57: PKG_ADD_OPTIONS: unbound variable The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: I0e3b1ed5aa9b2e624d9afbe46a8a344f88ddcc56 Reviewed-on: https://go-review.googlesource.com/c/143458 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-19 22:50:28 +03:00
Notes: "OpenBSD 6.4; GCE VM is built from script in build/env/openbsd-386",
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
SSHUsername: "gopher",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-freebsd-93-gce": &HostConfig{
VMImage: "freebsd-amd64-gce93",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
"host-freebsd-10_3": &HostConfig{
dashboard: rebuild FreeBSD VM to avoid 10 second boot prompt FreeBSD VMs boot up 10 seconds faster now, and shut down faster. Updates golang/go#22894 Updates golang/go#22895 Change-Id: I8eef979fc43d95d3c03104a761e70ec5737a3a97 Reviewed-on: https://go-review.googlesource.com/80304 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-29 01:07:29 +03:00
VMImage: "freebsd-amd64-103-b",
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
Notes: "FreeBSD 10.3; GCE VM is built from script in build/env/freebsd-amd64",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64", // TODO(bradfitz): why was this http instead of https?
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
env: []string{"CC=clang"},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
"host-freebsd-10_4": &HostConfig{
VMImage: "freebsd-amd64-104",
Notes: "FreeBSD 10.4; GCE VM is built from script in build/env/freebsd-amd64",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
SSHUsername: "gopher",
},
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
"host-freebsd-11_1": &HostConfig{
dashboard: rebuild FreeBSD VM to avoid 10 second boot prompt FreeBSD VMs boot up 10 seconds faster now, and shut down faster. Updates golang/go#22894 Updates golang/go#22895 Change-Id: I8eef979fc43d95d3c03104a761e70ec5737a3a97 Reviewed-on: https://go-review.googlesource.com/80304 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-29 01:07:29 +03:00
VMImage: "freebsd-amd64-111-b",
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
Notes: "FreeBSD 11.1; GCE VM is built from script in build/env/freebsd-amd64",
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64", // TODO(bradfitz): why was this http instead of https?
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
env: []string{"CC=clang"},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
},
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
"host-freebsd-11_2": &HostConfig{
VMImage: "freebsd-amd64-112",
Notes: "FreeBSD 11.2; GCE VM is built from script in build/env/freebsd-amd64",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
SSHUsername: "gopher",
},
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
"host-freebsd-12_0": &HostConfig{
env/freebsd-amd64: update FreeBSD 12.0 to a -RELEASE image Install GDB from the ports collection, the in-tree was removed from the base system in FreeBSD 12.0, and the older releases have an unsupported (6.1.1 or older version). Updates golang/go#25289 Change-Id: I8899a2a70c0b9ee9171621206431ca24a7e97704 Reviewed-on: https://go-review.googlesource.com/c/155977 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-12-29 20:53:25 +03:00
VMImage: "freebsd-amd64-120-v1",
Notes: "FreeBSD 12.0; GCE VM is built from script in build/env/freebsd-amd64",
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
SSHUsername: "gopher",
},
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
"host-netbsd-amd64-8_0": &HostConfig{
VMImage: "netbsd-amd64-8-0-2018q1",
Notes: "NetBSD 8.0RC1; GCE VM is built from script in build/env/netbsd-amd64",
dashboard: bump netbsd-amd64 bootstrap, enlarge VMs, remove 386 GOHOSTARCH Prepared using https://golang.org/cl/80895 Updates golang/go#20852 Change-Id: I186f74010f0a8be66eef5c369be664abe3f68f7f Reviewed-on: https://go-review.googlesource.com/80896 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-30 05:32:26 +03:00
machineType: "n1-highcpu-4",
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.netbsd-amd64",
dashboard: bump netbsd-amd64 bootstrap, enlarge VMs, remove 386 GOHOSTARCH Prepared using https://golang.org/cl/80895 Updates golang/go#20852 Change-Id: I186f74010f0a8be66eef5c369be664abe3f68f7f Reviewed-on: https://go-review.googlesource.com/80896 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-30 05:32:26 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-netbsd-amd64-2da6b33.tar.gz",
dashboard, cmd/buildlet, env/netbsd: enable netbsd ssh, re-disable netbsd-386 Fixes golang/go#22872 Updates golang/go#20852 Change-Id: I2b3454cb59a434cb12b10bb93cb4dd21d5187281 Reviewed-on: https://go-review.googlesource.com/81356 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-12-01 09:55:37 +03:00
SSHUsername: "root",
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
},
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
// Note: the netbsd-386 host hangs during the ../test phase of all.bash,
// so we don't use this for now. (See the netbsd-386-8 BuildConfig below.)
"host-netbsd-386-8_0": &HostConfig{
VMImage: "netbsd-386-8-0-2018q1",
Notes: "NetBSD 8.0RC1; GCE VM is built from script in build/env/netbsd-386",
dashboard: bump netbsd-amd64 bootstrap, enlarge VMs, remove 386 GOHOSTARCH Prepared using https://golang.org/cl/80895 Updates golang/go#20852 Change-Id: I186f74010f0a8be66eef5c369be664abe3f68f7f Reviewed-on: https://go-review.googlesource.com/80896 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-30 05:32:26 +03:00
machineType: "n1-highcpu-4",
all: start to revive netbsd-386 support It's still busted, but this will let other people debug. Updates golang/go#22914 Updates golang/go#20852 Updates golang/go#19339 Change-Id: Iedc21417e56418dab6abd433574ffef012ae43f3 Reviewed-on: https://go-review.googlesource.com/80435 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-29 03:44:46 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.netbsd-386",
dashboard: re-disable netbsd/386 builder until binary packages are built Also update the bootstrap toolchain (verified in VMware VM locally). Updates golang/go#20852 Change-Id: Iac875419ed8dc51fb9db09d8948067f24c601e17 Reviewed-on: https://go-review.googlesource.com/82018 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-12-06 02:39:30 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-netbsd-386-0b3b511.tar.gz",
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
SSHUsername: "root",
all: start to revive netbsd-386 support It's still busted, but this will let other people debug. Updates golang/go#22914 Updates golang/go#20852 Updates golang/go#19339 Change-Id: Iedc21417e56418dab6abd433574ffef012ae43f3 Reviewed-on: https://go-review.googlesource.com/80435 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-29 03:44:46 +03:00
},
dashboard: add netbsd-arm-bsiegert This seems to be the last builder still using the old builder binary and not buildlet. It needs an entry in here to switch to buildlet and stage0. Change-Id: If2fdd2298ab88d56b6850061050caf27ca7c8983 Reviewed-on: https://go-review.googlesource.com/c/build/+/168061 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-19 16:28:04 +03:00
"host-netbsd-arm-bsiegert": &HostConfig{
IsReverse: true,
ExpectNum: 1,
env: []string{"GOROOT_BOOTSTRAP=/usr/pkg/go112"},
OwnerGithub: "bsiegert",
},
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
"host-dragonfly-amd64-tdfbsd": &HostConfig{
IsReverse: true,
ExpectNum: 1,
dashboard: set GOROOT_BOOTSTRAP for dragonfly-amd64 and freebsd-arm Updates golang/go#21191 Change-Id: I09854a99c17dec204ee3a1f7c197bb7b69895979 Reviewed-on: https://go-review.googlesource.com/53491 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 02:14:51 +03:00
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
ReverseAliases: []string{"dragonfly-amd64"},
OwnerGithub: "tdfbsd",
},
"host-freebsd-arm-paulzhol": &HostConfig{
IsReverse: true,
ExpectNum: 1,
dashboard: add freebsd-arm host notes Copied from https://github.com/golang/go/wiki/DashboardBuilders Change-Id: If69e5fc85af380bf92f408c68b481d0cd5cc1bdb Reviewed-on: https://go-review.googlesource.com/53359 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 06:57:03 +03:00
Notes: "Cubiboard2 1Gb RAM dual-core Cortex-A7 (Allwinner A20), FreeBSD 11.1-RELEASE",
dashboard: set GOROOT_BOOTSTRAP for dragonfly-amd64 and freebsd-arm Updates golang/go#21191 Change-Id: I09854a99c17dec204ee3a1f7c197bb7b69895979 Reviewed-on: https://go-review.googlesource.com/53491 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 02:14:51 +03:00
env: []string{"GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4"},
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
ReverseAliases: []string{"freebsd-arm-paulzhol"},
OwnerGithub: "paulzhol",
},
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
"host-plan9-arm-0intro": &HostConfig{
IsReverse: true,
ExpectNum: 1,
Notes: "Raspberry Pi 3 Model B, Plan 9 from Bell Labs",
OwnerGithub: "0intro",
},
"host-plan9-amd64-0intro": &HostConfig{
IsReverse: true,
ExpectNum: 1,
OwnerGithub: "0intro",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-plan9-386-gce": &HostConfig{
env/plan9: revert SMP changes, stop running cmd/go tests Revert "env: enable SMP in Plan 9 image" This reverts commit 6478d3e7bdff4c8d810ba44f970ca4b0bfc92405. Revert "dashboard: bump plan9 GCE image" This reverts commit ce208996bf954a87d68789bb6a9000256f016c77. Updates golang/go#29822 Change-Id: I47e75d135e092ee9d5a80c66e91c1048c67cd5f6 Reviewed-on: https://go-review.googlesource.com/c/158542 Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: David du Colombier <0intro@gmail.com>
2019-01-19 01:19:58 +03:00
VMImage: "plan9-386-v7",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "Plan 9 from 0intro; GCE VM is built from script in build/env/plan9-386",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.plan9-386",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-plan9-386.tar.gz",
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// We *were* using n1-standard-1 because Plan 9 can only
// reliably use a single CPU. Using 2 or 4 and we see
// test failures. See:
// https://golang.org/issue/8393
// https://golang.org/issue/9491
// n1-standard-1 has 3.6 GB of memory which WAS (see below)
// overkill (userspace probably only sees 2GB anyway),
// but it's the cheapest option. And plenty to keep
// our ~250 MB of inputs+outputs in its ramfs.
//
// But the docs says "For the n1 series of machine
// types, a virtual CPU is implemented as a single
// hyperthread on a 2.6GHz Intel Sandy Bridge Xeon or
// Intel Ivy Bridge Xeon (or newer) processor. This
// means that the n1-standard-2 machine type will see
// a whole physical core."
//
// ... so we used n1-highcpu-2 (1.80 RAM, still
// plenty), just so we can get 1 whole core for the
// single-core Plan 9. It will see 2 virtual cores and
// only use 1, but we hope that 1 will be more powerful
// and we'll stop timing out on tests.
machineType: "n1-highcpu-4",
dashboard: increase plan9's GO_TEST_TIMEOUT_SCALE Updates golang/go#29801 Change-Id: Ib346d717ae9b8de18666d6ea543d9bdcd2856f3b Reviewed-on: https://go-review.googlesource.com/c/158537 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-01-18 19:14:12 +03:00
env: []string{"GO_TEST_TIMEOUT_SCALE=3"},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
"host-windows-amd64-2008": &HostConfig{
env/windows: disable windows defender on builders Windows Defender scans all new files written to disk which slows down build performance. - Uninstall Windows Defender as part of image building - Refactor/Cleanup registry section for consistency - Promote new image version to dashboard (built in staging/prod) fixes golang/go#26055 Change-Id: I3e16b9a9581824c20abed5e8ffae1efd46c6dd09 Reviewed-on: https://go-review.googlesource.com/121937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-02 20:49:55 +03:00
VMImage: "windows-amd64-server-2008r2-v7",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
dashboard,cmd/buildlet,cmd/coordinator: public key auth on windows Enable public key auth for windows buildlets. Previously the username/password was hardcoded. This change enables the `/connect-ssh` http endpoint on the buildlet to setup authorized_keys and uses the coordinator to proxy ssh traffic to the buildlet. Validation: - run local coordinator (with patches to dev env to act like GCE) with changes - build windows buildlet, post to own bucket - execute: gomote -localdev create windows-amd64-2012 ssh -p 2222 user-jrjohnson-windows-amd64-2012-0@127.0.0.1 Fixes golang/go#21247 Change-Id: If7adc7e6296a071d50412d9fa9d46ecd9e6cc3f3 Reviewed-on: https://go-review.googlesource.com/52350 Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-08-01 02:55:28 +03:00
SSHUsername: "gopher",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
},
"host-windows-amd64-2012": &HostConfig{
env/windows: disable windows defender on builders Windows Defender scans all new files written to disk which slows down build performance. - Uninstall Windows Defender as part of image building - Refactor/Cleanup registry section for consistency - Promote new image version to dashboard (built in staging/prod) fixes golang/go#26055 Change-Id: I3e16b9a9581824c20abed5e8ffae1efd46c6dd09 Reviewed-on: https://go-review.googlesource.com/121937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-02 20:49:55 +03:00
VMImage: "windows-amd64-server-2012r2-v7",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
dashboard,cmd/buildlet,cmd/coordinator: public key auth on windows Enable public key auth for windows buildlets. Previously the username/password was hardcoded. This change enables the `/connect-ssh` http endpoint on the buildlet to setup authorized_keys and uses the coordinator to proxy ssh traffic to the buildlet. Validation: - run local coordinator (with patches to dev env to act like GCE) with changes - build windows buildlet, post to own bucket - execute: gomote -localdev create windows-amd64-2012 ssh -p 2222 user-jrjohnson-windows-amd64-2012-0@127.0.0.1 Fixes golang/go#21247 Change-Id: If7adc7e6296a071d50412d9fa9d46ecd9e6cc3f3 Reviewed-on: https://go-review.googlesource.com/52350 Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-08-01 02:55:28 +03:00
SSHUsername: "gopher",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
},
"host-windows-amd64-2016": &HostConfig{
env/windows: disable windows defender on builders Windows Defender scans all new files written to disk which slows down build performance. - Uninstall Windows Defender as part of image building - Refactor/Cleanup registry section for consistency - Promote new image version to dashboard (built in staging/prod) fixes golang/go#26055 Change-Id: I3e16b9a9581824c20abed5e8ffae1efd46c6dd09 Reviewed-on: https://go-review.googlesource.com/121937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-02 20:49:55 +03:00
VMImage: "windows-amd64-server-2016-v7",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
dashboard,cmd/buildlet,cmd/coordinator: public key auth on windows Enable public key auth for windows buildlets. Previously the username/password was hardcoded. This change enables the `/connect-ssh` http endpoint on the buildlet to setup authorized_keys and uses the coordinator to proxy ssh traffic to the buildlet. Validation: - run local coordinator (with patches to dev env to act like GCE) with changes - build windows buildlet, post to own bucket - execute: gomote -localdev create windows-amd64-2012 ssh -p 2222 user-jrjohnson-windows-amd64-2012-0@127.0.0.1 Fixes golang/go#21247 Change-Id: If7adc7e6296a071d50412d9fa9d46ecd9e6cc3f3 Reviewed-on: https://go-review.googlesource.com/52350 Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-08-01 02:55:28 +03:00
SSHUsername: "gopher",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
},
dashboard: add windows-arm builder Updates golang/go#26148 Change-Id: I23a2c7bd1c2f8f7c960aaedfe0dcdee4e11825ab Reviewed-on: https://go-review.googlesource.com/125643 Reviewed-by: Andrew Bonventre <andybons@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-24 22:59:34 +03:00
"host-windows-arm-iotcore": &HostConfig{
IsReverse: true,
ExpectNum: 1,
OwnerGithub: "jordanrh1",
env: []string{"GOROOT_BOOTSTRAP=C:\\Data\\Go"},
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-darwin-10_8": &HostConfig{
IsReverse: true,
dashboard: adjust how many Mac VMs we expect Updates golang/go#23859 Change-Id: I7e0fed5b17430669a726a8a7bbf8d3efc190034c Reviewed-on: https://go-review.googlesource.com/95735 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-02-21 04:25:11 +03:00
ExpectNum: 0,
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "MacStadium OS X 10.8 VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ReverseAliases: []string{"darwin-amd64-10_8"},
SSHUsername: "gopher",
HermeticReverse: false, // TODO: make it so, like 10.12
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
"host-darwin-10_10": &HostConfig{
IsReverse: true,
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
ExpectNum: 3,
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "MacStadium OS X 10.10 VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ReverseAliases: []string{"darwin-amd64-10_10"},
SSHUsername: "gopher",
HermeticReverse: false, // TODO: make it so, like 10.12
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
"host-darwin-10_11": &HostConfig{
IsReverse: true,
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
ExpectNum: 7,
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "MacStadium OS X 10.11 VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ReverseAliases: []string{"darwin-amd64-10_11"},
SSHUsername: "gopher",
HermeticReverse: false, // TODO: make it so, like 10.12
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
"host-darwin-10_12": &HostConfig{
IsReverse: true,
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
ExpectNum: 3,
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
Notes: "MacStadium OS X 10.12 VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ReverseAliases: []string{"darwin-amd64-10_12"},
SSHUsername: "gopher",
HermeticReverse: true, // we destroy the VM when done & let cmd/makemac recreate
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
},
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
"host-darwin-10_14": &HostConfig{
IsReverse: true,
ExpectNum: 7,
Notes: "MacStadium macOS Mojave (10.14) VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/goboot", // Go 1.12.1
},
SSHUsername: "gopher",
HermeticReverse: true, // we destroy the VM when done & let cmd/makemac recreate
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-linux-s390x": &HostConfig{
Notes: "run by IBM",
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
OwnerGithub: "mundaym",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
IsReverse: true,
env: []string{"GOROOT_BOOTSTRAP=/var/buildlet/go-linux-s390x-bootstrap"},
ReverseAliases: []string{"linux-s390x-ibm"},
},
all: add ppc64 (big endian) builders Change-Id: I0fddda33179555a50b6f03dcff13338544b742c3 Reviewed-on: https://go-review.googlesource.com/30599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-06 23:30:58 +03:00
"host-linux-ppc64-osu": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
Notes: "Debian jessie; run by Go team on osuosl.org",
IsReverse: true,
ExpectNum: 5,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
ReverseAliases: []string{"linux-ppc64-buildlet"},
SSHUsername: "debian",
HermeticReverse: false, // TODO: use rundockerbuildlet like arm64
all: add ppc64 (big endian) builders Change-Id: I0fddda33179555a50b6f03dcff13338544b742c3 Reviewed-on: https://go-review.googlesource.com/30599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-06 23:30:58 +03:00
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-linux-ppc64le-osu": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
Notes: "Debian jessie; run by Go team on osuosl.org",
IsReverse: true,
ExpectNum: 5,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
ReverseAliases: []string{"linux-ppc64le-buildlet"},
SSHUsername: "debian",
HermeticReverse: false, // TODO: use rundockerbuildlet like arm64
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard: add linux-ppc64le POWER9 builder This change adds an extra linux-ppc64le builder running on POWER9. Fixes golang/go#30138 Change-Id: I23bdfc116bb89c6d96be330c2c87827c47705b6b Reviewed-on: https://go-review.googlesource.com/c/163757 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:05:12 +03:00
"host-linux-ppc64le-power9-osu": &HostConfig{
Notes: "Debian jessie; run by IBM",
OwnerGithub: "ceseo",
IsReverse: true,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
ReverseAliases: []string{"linux-ppc64le-power9osu"},
SSHUsername: "debian",
HermeticReverse: false, // TODO: use rundockerbuildlet like arm64
},
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
"host-linux-arm64-packet": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
Notes: "On 96 core packet.net host (Xenial) in Docker containers (Jessie); run by Go team. See x/build/env/linux-arm64/packet",
IsReverse: true,
HermeticReverse: true,
ExpectNum: 20,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
SSHUsername: "root",
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-solaris-amd64": &HostConfig{
Notes: "run by Go team on Joyent, on a SmartOS 'infrastructure container'",
IsReverse: true,
cmd/coordinator: clean up reverse buildlet code, export status JSON Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-24 20:50:49 +03:00
ExpectNum: 5,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
env: []string{"GOROOT_BOOTSTRAP=/root/go-solaris-amd64-bootstrap", "HOME=/root"},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
ReverseAliases: []string{"solaris-amd64-smartosbuildlet"},
},
x/build: add Solaris amd64 release builder to dashboard Fixes golang/go#20949 Change-Id: I4f85ee76783a1240a2ebd4a2a1f167ab849fcebe Reviewed-on: https://go-review.googlesource.com/47811 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-07 23:41:20 +03:00
"host-solaris-oracle-amd64-oraclerel": &HostConfig{
Notes: "Oracle Solaris amd64 Release System",
dashboard: remove a stale solaris builder and old owner information on another Change-Id: I3997b9ceb4981fd1cf538dcb0b3d82fd5e885e8d Reviewed-on: https://go-review.googlesource.com/c/build/+/170439 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-02 22:10:26 +03:00
Owner: "", // TODO: find current owner
OwnerGithub: "", // TODO: find current owner
dashboard: add Oracle Solaris development builder Fixes golang/go#15072 Change-Id: I85677636a0f7f8b6a7b2405bb0669c39cc1da7f0 Reviewed-on: https://go-review.googlesource.com/46831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-27 20:11:15 +03:00
IsReverse: true,
ExpectNum: 1,
x/build: bootstrap tweaks for solaris builders Change bootstrap logic for solaris builders to avoid applying it to Oracle Solaris builders. Additionally, update the location of the Go bootstrap to reflect actual Oracle Solaris builder configuration. Fixes golang/go#21241 Change-Id: Ia2f21fab1581b4c195256c0dca7a87b4212798bf Reviewed-on: https://go-review.googlesource.com/52310 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-31 23:58:19 +03:00
env: []string{"GOROOT_BOOTSTRAP=/opt/golang/go-solaris-amd64-bootstrap"},
dashboard: add Oracle Solaris development builder Fixes golang/go#15072 Change-Id: I85677636a0f7f8b6a7b2405bb0669c39cc1da7f0 Reviewed-on: https://go-review.googlesource.com/46831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-27 20:11:15 +03:00
},
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
"host-linux-mips": &HostConfig{
dashboard: update stale ownership info for now-dead MIPS builders Updates golang/go#31217 Change-Id: I9b8e64c24ed303166e609c13fa4b69d83ddcd0c1 Reviewed-on: https://go-review.googlesource.com/c/build/+/170444 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-03 01:01:16 +03:00
Notes: "", // once ran by Brendan Kirby (@MIPSbkirby), imgtec.com; email bounces
OwnerGithub: "",
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
IsReverse: true,
ExpectNum: 1,
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mips",
"GOARCH=mips",
"GOHOSTARCH=mips",
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
"GO_TEST_TIMEOUT_SCALE=4",
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
},
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
ReverseAliases: []string{"linux-mips"},
},
"host-linux-mipsle": &HostConfig{
dashboard: update stale ownership info for now-dead MIPS builders Updates golang/go#31217 Change-Id: I9b8e64c24ed303166e609c13fa4b69d83ddcd0c1 Reviewed-on: https://go-review.googlesource.com/c/build/+/170444 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-03 01:01:16 +03:00
Notes: "", // once ran by Brendan Kirby (@MIPSbkirby), imgtec.com; email bounces
OwnerGithub: "",
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
IsReverse: true,
ExpectNum: 1,
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mipsle",
"GOARCH=mipsle",
"GOHOSTARCH=mipsle",
},
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
ReverseAliases: []string{"linux-mipsle"},
},
"host-linux-mips64": &HostConfig{
dashboard: update stale ownership info for now-dead MIPS builders Updates golang/go#31217 Change-Id: I9b8e64c24ed303166e609c13fa4b69d83ddcd0c1 Reviewed-on: https://go-review.googlesource.com/c/build/+/170444 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-03 01:01:16 +03:00
Notes: "", // once ran by Brendan Kirby (@MIPSbkirby), imgtec.com; email bounces
OwnerGithub: "",
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
IsReverse: true,
ExpectNum: 1,
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mips64",
"GOARCH=mips64",
"GOHOSTARCH=mips64",
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
"GO_TEST_TIMEOUT_SCALE=4",
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
},
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
ReverseAliases: []string{"linux-mips64"},
},
"host-linux-mips64le": &HostConfig{
dashboard: update stale ownership info for now-dead MIPS builders Updates golang/go#31217 Change-Id: I9b8e64c24ed303166e609c13fa4b69d83ddcd0c1 Reviewed-on: https://go-review.googlesource.com/c/build/+/170444 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-03 01:01:16 +03:00
Notes: "", // once ran by Brendan Kirby (@MIPSbkirby), imgtec.com; email bounces
OwnerGithub: "",
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
IsReverse: true,
ExpectNum: 1,
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mips64le",
"GOARCH=mips64le",
"GOHOSTARCH=mips64le",
},
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
ReverseAliases: []string{"linux-mips64le"},
},
dashboard: split mobile builders into iOS and Android Even though both mobile builders will run off the same host Mac Mini, Android and iOS builds can run simultaneously. Split the eliasnaur host in two to allow that. Change-Id: If3db4dadb56b8a4b212dcbb3015e3e6c1b69608d Reviewed-on: https://go-review.googlesource.com/39130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-31 11:08:01 +03:00
"host-darwin-amd64-eliasnaur-android": &HostConfig{
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
Notes: "Mac Mini hosted by Elias Naur, running the android reverse buildlet",
OwnerGithub: "eliasnaur",
IsReverse: true,
ExpectNum: 1,
dashboard: split mobile builders into iOS and Android Even though both mobile builders will run off the same host Mac Mini, Android and iOS builds can run simultaneously. Split the eliasnaur host in two to allow that. Change-Id: If3db4dadb56b8a4b212dcbb3015e3e6c1b69608d Reviewed-on: https://go-review.googlesource.com/39130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-31 11:08:01 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap",
"GOHOSTARCH=amd64",
"GOOS=android",
},
},
dashboard: add zenly as builder for ios (Iphone 5C, 7+) Change-Id: Ie1dd5c53ed0b2d207297ee0cd3ba46a111167e30 Reviewed-on: https://go-review.googlesource.com/125815 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-25 16:14:32 +03:00
"host-darwin-amd64-zenly-ios": &HostConfig{
Notes: "MacBook Pro hosted by Zenly, running the ios reverse buildlet",
OwnerGithub: "znly",
IsReverse: true,
ExpectNum: 1,
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/Cellar/1.10.3/libexec",
"GOHOSTARCH=amd64",
},
},
dashboard: add aix-ppc64 builder This is a starting point for adding an aix-ppc64 reverse builder. There can be further adjustments made to it as needed. Updates golang/go#27160. Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508 Reviewed-on: https://go-review.googlesource.com/131335 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-24 21:41:05 +03:00
"host-aix-ppc64-osuosl": &HostConfig{
Notes: "AIX 7.2 VM on OSU; run by Tony Reix",
OwnerGithub: "trex58",
IsReverse: true,
ExpectNum: 1,
env: []string{"GOROOT_BOOTSTRAP=/opt/freeware/lib/golang"},
},
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
"host-android-amd64-emu": &HostConfig{
Notes: "Debian Buster w/ Android SDK + emulator (use nested virt)",
env/android-amd64-emu: enable gomobile device tests By adding gradle and Android build tools to PATH, setting ANDROID_HOME, and adding the OpenAL headers to CGO_CFLAGS. Change-Id: Ifeaa1b82f4108afc3787eee4449b3317185bbaec Reviewed-on: https://go-review.googlesource.com/c/build/+/164658 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-01 13:21:20 +03:00
ContainerImage: "android-amd64-emu:bff27c0c9263",
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
KonletVMImage: "android-amd64-emu",
NestedVirt: true,
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
}
func init() {
for key, c := range Hosts {
if key == "" {
panic("empty string key in Hosts")
}
if c.HostType == "" {
c.HostType = key
}
if c.HostType != key {
panic(fmt.Sprintf("HostType %q != key %q", c.HostType, key))
}
nSet := 0
if c.VMImage != "" {
nSet++
}
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
if c.ContainerImage != "" {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
nSet++
}
if c.IsReverse {
nSet++
}
if nSet != 1 {
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
panic(fmt.Sprintf("exactly one of VMImage, ContainerImage, IsReverse must be set for host %q; got %v", key, nSet))
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
}
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
if c.buildletURLTmpl == "" && (c.VMImage != "" || c.ContainerImage != "") {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
panic(fmt.Sprintf("missing buildletURLTmpl for host type %q", key))
}
}
}
// A HostConfig describes the available ways to obtain buildlets of
// different types. Some host configs can server multiple
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
// builders. For example, a host config of "host-linux-jessie" can
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// serve linux-amd64, linux-amd64-race, linux-386, linux-386-387, etc.
type HostConfig struct {
// HostType is the unique name of this host config. It is also
// the key in the Hosts map.
HostType string
// buildletURLTmpl is the URL "template" ($BUCKET is auto-expanded)
// for the URL to the buildlet binary.
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
// This field is required for VM and Container builders. It's not
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// needed for reverse buildlets because in that case, the buildlets
// are already running and their stage0 should know how to update it
// it automatically.
buildletURLTmpl string
// Exactly 1 of these must be set:
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
VMImage string // e.g. "openbsd-amd64-60"
ContainerImage string // e.g. "linux-buildlet-std:latest" (suffix after "gcr.io/<PROJ>/")
IsReverse bool // if true, only use the reverse buildlet pool
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// GCE options, if VMImage != ""
buildlet, dashboard: add min CPU platform knob, require Skylake for OpenBSD I thought this would be enough for OpenBSD to select the TSC on its own without being forced to (as in CL 160319), but apparently it is not: https://github.com/golang/go/issues/29223#issuecomment-459035482 So it seems like we want both this CL and CL 160319. Updates golang/go#29223 Change-Id: I0a092d62881d8dcce0ef1129d8d32d8f4025b6ac Reviewed-on: https://go-review.googlesource.com/c/160457 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-30 20:13:07 +03:00
machineType string // optional GCE instance type
RegularDisk bool // if true, use spinning disk instead of SSD
MinCPUPlatform string // optional; https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
cmd/coordinator: clean up reverse buildlet code, export status JSON Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-24 20:50:49 +03:00
// ReverseOptions:
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ExpectNum int // expected number of reverse buildlets of this type
HermeticReverse bool // whether reverse buildlet has fresh env per conn
cmd/coordinator: clean up reverse buildlet code, export status JSON Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-24 20:50:49 +03:00
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
// Container image options, if ContainerImage != "":
env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx This adds scripts to create a new builder host image that acts like Container-Optimized OS (has docker, runs konlet on startup) but with a Debian 9 kernel + userspace that permits KVM for nested virtualization. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884 Reviewed-on: https://go-review.googlesource.com/c/163301 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-22 00:10:54 +03:00
NestedVirt bool // container requires VMX nested virtualization
KonletVMImage string // optional VM image (containing konlet) to use instead of default
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// Optional base env. GOROOT_BOOTSTRAP should go here if the buildlet
// has Go 1.4+ baked in somewhere.
env []string
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
// These template URLs may contain $BUCKET which is expanded to the
// relevant Cloud Storage bucket as specified by the build environment.
goBootstrapURLTmpl string // optional URL to a built Go 1.4+ tar.gz
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
Owner string // optional email of owner; "bradfitz@golang.org", empty means golang-dev
OwnerGithub string // optional GitHub username of owner
Notes string // notes for humans
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername string // username to ssh as, empty means not supported
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// ReverseAliases lists alternate names for this buildlet
// config, for older clients doing a reverse dial into the
// coordinator from outside. This prevents us from updating
// 75+ dedicated machines/VMs atomically, switching them to
// the new "host-*" names.
// This is only applicable if IsReverse.
ReverseAliases []string
}
// A BuildConfig describes how to run a builder.
type BuildConfig struct {
// Name is the unique name of the builder, in the form of
// "GOOS-GOARCH" or "GOOS-GOARCH-suffix". For example,
// "darwin-386", "linux-386-387", "linux-amd64-race". Some
// suffixes are well-known and carry special meaning, such as
// "-race".
Name string
// HostType is the required key into the Hosts map, describing
// the type of host this build will run on.
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
// For example, "host-linux-jessie".
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
HostType string
Notes string // notes for humans
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// tryBot optionally specifies a policy func for whether trybots are enabled.
// nil means off. Even if tryBot returns true, BuildConfig.BuildsRepo must also
// return true. See the implementation of BuildConfig.BuildsRepoTryBot.
// The proj is "go", "net", etc. The branch is proj's branch.
// The goBranch is the same as branch for proj "go", else it's the go branch
// ("master, "release-branch.go1.12", etc).
tryBot func(proj, branch, goBranch string) bool
tryOnly bool // only used for trybots, and not regular builds
CompileOnly bool // if true, compile tests, but don't run them
FlakyNet bool // network tests are flaky (try anyway, but ignore some failures)
// buildsRepo optionally specifies whether this
// builder does builds (of any type) for the given repo ("go",
// "net", etc) and its branch ("master", "release-branch.go1.12").
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
// If nil, a default policy is used. (see buildsRepoAtAll for details)
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// goBranch is the branch of "go" to build against. If repo == "go",
// goBranch == branch.
buildsRepo func(repo, branch, goBranch string) bool
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
// MinimumGoVersion optionally specifies the minimum Go version
// this builder is allowed to use. It can be useful for skipping
// builders that are too new and no longer support some supported
// Go versions. It doesn't need to be set for builders that support
// all supported Go versions.
//
// Note: This field currently has effect on trybot runs only.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
//
// TODO: unexport this and make buildsRepoAtAll return false on too-old
// of repos. The callers in coordinator will need updating.
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
MinimumGoVersion types.MajorMinor
cmd/coordinator: support limiting number of builds at once Change-Id: Icd0054f2b943abf2e8a15c32d358ff06541f75f7 Reviewed-on: https://go-review.googlesource.com/41130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-19 23:45:24 +03:00
// MaxAtOnce optionally specifies a cap of how many builds of
// this type can run at once. Zero means unlimited. This is a
// temporary measure until the build scheduler
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
// (golang.org/issue/19178) is done, at which point this field
// should be deleted.
cmd/coordinator: support limiting number of builds at once Change-Id: Icd0054f2b943abf2e8a15c32d358ff06541f75f7 Reviewed-on: https://go-review.googlesource.com/41130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-19 23:45:24 +03:00
MaxAtOnce int
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
// SkipSnapshot, if true, means to not fetch a tarball
// snapshot of the world post-make.bash from the buildlet (and
// thus to not write it to Google Cloud Storage). This is
// incompatible with sharded tests, and should only be used
// for very slow builders or networks, unable to transfer
// the tarball in under ~5 minutes.
SkipSnapshot bool
cmd/coordinator: run benchmarks on try work Benchmarks are treated as unit tests and distributed to the test helpers, which allows them to fit in our 5m trybot budget. Currently we only run the go1 and x/benchmarks. Running package benchmarks is a TODO. This feature is disabled by default, and is enabled by the "farmer-run-bench" project attribute. Updates golang/go#19178 Updates golang/go#19871 Change-Id: I9c3a14da60c3662e7e2cb4e71953060915cc4364 Reviewed-on: https://go-review.googlesource.com/38306 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-15 23:18:30 +03:00
// RunBench causes the coordinator to run benchmarks on this buildlet type.
RunBench bool
dashboard: add a make.bash-only builder on real ARM hardware This is a new builder in prep for the change to the "linux-arm" builder where the GOARCH=arm make.bash will be cross-compiled from a Kubernetes container on fast hardware. Updates golang/go#17105 (cross-compile ARM builders' make.bash) Updates golang/go#17104 (5 minute trybots) Change-Id: Icfd2644d77639f731151abe54839322960418254 Reviewed-on: https://go-review.googlesource.com/29670 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 02:29:31 +03:00
// StopAfterMake causes the build to stop after the make
// script completes, returning its result as the result of the
// whole build. It does not run or compile any of the tests,
// nor does it write a snapshot of the world to cloud
// storage. This option is only supported for builders whose
// BuildConfig.SplitMakeRun returns true.
StopAfterMake bool
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// needsGoProxy is whether this builder should have GOPROXY set.
// Currently this is only for the longtest builder, which needs
// to run cmd/go tests fetching from the network.
needsGoProxy bool
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
// InstallRacePackages controls which packages to "go install
// -race <pkgs>" after running make.bash (or equivalent). If
// the builder ends in "-race", the default if non-nil is just
// "std".
InstallRacePackages []string
cmd/coordinator: move git ancestor checking before buildlet creation, generalize This also adds GoDeps to the builders config file and adds a dependency for the ssacheck builder, which currently fails on older release branches with: > Failed on linux-amd64-ssacheck: > https://storage.googleapis.com/go-build-log/1255edaf/linux-amd64-ssacheck_18f7e015.log > > ... > compile: unknown debug key -d dclstack Change-Id: I4ad0a298705feed56f467f3c8d9488f8be2e4af5 Reviewed-on: https://go-review.googlesource.com/43774 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-20 00:23:14 +03:00
// GoDeps is a list of of git sha1 commits that must be in the
// commit to be tested's history. If absent, this builder is
// not run for that commit.
GoDeps []string
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
// shouldRunDistTest optionally specifies a function to
// override the BuildConfig.ShouldRunDistTest method's
// default behavior.
shouldRunDistTest func(distTest string, isTry bool) bool
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
// numTestHelpers is the number of _additional_ buildlets
// past the first one to help out with sharded tests.
// For trybots, the numTryHelpers value is used, unless it's
// zero, in which case numTestHelpers is used.
numTestHelpers int
numTryTestHelpers int // for trybots. if 0, numTesthelpers is used
cmd/coordinator: test sharding, better status, logs Also gomote updates which came about during the process of developing and debugging this. Change-Id: Ia53d674118a6b99bcdda7062d3b7161279b6ad52 Reviewed-on: https://go-review.googlesource.com/10463 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-28 07:51:25 +03:00
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
env []string // extra environment ("key=value") pairs
allScriptArgs []string
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
testHostConf *HostConfig // override HostConfig for testing, at least for now
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// Env returns the environment variables this builder should run with.
cmd/coordinator: set GO_BUILDER_NAME env var when exec'ing on buildlet This will allow tests to identify that they are running on the continuous build infrastructure and run extra and/or different tests. Fixes golang/go#11346 Change-Id: Iafe25babc764b7e012c05f0b62331b11690bf194 Reviewed-on: https://go-review.googlesource.com/13120 Reviewed-by: Russ Cox <rsc@golang.org>
2015-08-04 05:19:02 +03:00
func (c *BuildConfig) Env() []string {
dashboard: mark some builders as having flaky networks Follow-up to https://golang.org/cl/22852 Change-Id: I5f0cd25248a8892b7d1b0f14e87b12b8b9a8da3a Reviewed-on: https://go-review.googlesource.com/22883 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2016-05-06 21:55:26 +03:00
env := []string{"GO_BUILDER_NAME=" + c.Name}
if c.FlakyNet {
env = append(env, "GO_BUILDER_FLAKY_NET=1")
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
env = append(env, c.hostConf().env...)
dashboard: mark some builders as having flaky networks Follow-up to https://golang.org/cl/22852 Change-Id: I5f0cd25248a8892b7d1b0f14e87b12b8b9a8da3a Reviewed-on: https://go-review.googlesource.com/22883 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2016-05-06 21:55:26 +03:00
return append(env, c.env...)
cmd/coordinator: set GO_BUILDER_NAME env var when exec'ing on buildlet This will allow tests to identify that they are running on the continuous build infrastructure and run extra and/or different tests. Fixes golang/go#11346 Change-Id: Iafe25babc764b7e012c05f0b62331b11690bf194 Reviewed-on: https://go-review.googlesource.com/13120 Reviewed-by: Russ Cox <rsc@golang.org>
2015-08-04 05:19:02 +03:00
}
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// ModulesEnv returns the extra module-specific environment variables
// to append to this builder as a function of the repo being built
// ("go", "oauth2", "net", etc).
func (c *BuildConfig) ModulesEnv(repo string) (env []string) {
if c.IsReverse() && repo != "go" {
env = append(env, "GO_BUILDER_SET_GOPROXY=coordinator")
}
switch repo {
dashboard: set GOPROXY=off for the "go" repo to catch more network usage Updates golang/go#30612 Updates golang/go#30760 Change-Id: I296f4f7a163c1bdd59e839229c0961b21f26da89 Reviewed-on: https://go-review.googlesource.com/c/build/+/167039 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 07:44:17 +03:00
case "go":
if !c.OutboundNetworkAllowed() {
env = append(env, "GOPROXY=off")
}
dashboard: set GO111MODULE=on for x/website tests The x/website repository uses go.mod to specify precise versions of its dependencies, and therefore requires the use of GO111MODULE=on to build correctly. So set GO111MODULE=on when running its tests. Change-Id: I4acc80fd207e257aa77dc0e296b1039c83fa3d2a Reviewed-on: https://go-review.googlesource.com/c/build/+/165743 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-06 21:47:35 +03:00
case "oauth2", "build", "website":
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
env = append(env, "GO111MODULE=on")
}
return
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *BuildConfig) IsReverse() bool { return c.hostConf().IsReverse }
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
func (c *BuildConfig) IsContainer() bool { return c.hostConf().IsContainer() }
func (c *HostConfig) IsContainer() bool { return c.ContainerImage != "" }
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
func (c *BuildConfig) IsVM() bool { return c.hostConf().IsVM() }
func (c *HostConfig) IsVM() bool { return c.VMImage != "" }
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
dashboard: create buildlet client package, move coordinator code into it Operation Packification, step 2 of tons. Eventually the buildlet client binary will use this stuff now. Change-Id: I4cf5f3e6beb9e56bdc795ed513ce6daaf61425e3 Reviewed-on: https://go-review.googlesource.com/2921 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 03:29:16 +03:00
func (c *BuildConfig) GOOS() string { return c.Name[:strings.Index(c.Name, "-")] }
func (c *BuildConfig) GOARCH() string {
arch := c.Name[strings.Index(c.Name, "-")+1:]
i := strings.Index(arch, "-")
if i == -1 {
return arch
}
return arch[:i]
}
cmd/coordinator: test sharding, better status, logs Also gomote updates which came about during the process of developing and debugging this. Change-Id: Ia53d674118a6b99bcdda7062d3b7161279b6ad52 Reviewed-on: https://go-review.googlesource.com/10463 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-28 07:51:25 +03:00
// FilePathJoin is mostly like filepath.Join (without the cleaning) except
// it uses the path separator of c.GOOS instead of the host system's.
func (c *BuildConfig) FilePathJoin(x ...string) string {
if c.GOOS() == "windows" {
return strings.Join(x, "\\")
}
return strings.Join(x, "/")
}
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
// DistTestsExecTimeout returns how long the coordinator should wait
// for a cmd/dist test execution to run the provided dist test names.
func (c *BuildConfig) DistTestsExecTimeout(distTests []string) time.Duration {
// TODO: consider using distTests? We never did before, but
// now we have the TestStats in the coordinator. Pass in a
// *buildstats.TestStats and use historical data times some
// fudge factor? For now just use the old 20 minute limit
// we've used since 2014, but scale it by the
// GO_TEST_TIMEOUT_SCALE for the super slow builders which
// struggle with, say, the cgo tests. (which should be broken
// up into separate dist tests or shards, like the test/ dir
// was)
d := 20 * time.Minute
d *= time.Duration(c.timeoutScale())
return d
}
// timeoutScale returns this builder's GO_TEST_TIMEOUT_SCALE value, or 1.
func (c *BuildConfig) timeoutScale() int {
const pfx = "GO_TEST_TIMEOUT_SCALE="
for _, env := range [][]string{c.env, c.hostConf().env} {
for _, kv := range env {
if strings.HasPrefix(kv, pfx) {
if n, err := strconv.Atoi(kv[len(pfx):]); err == nil && n > 0 {
return n
}
}
}
}
return 1
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *BuildConfig) hostConf() *HostConfig {
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
if c.testHostConf != nil {
return c.testHostConf
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
if c, ok := Hosts[c.HostType]; ok {
return c
}
panic(fmt.Sprintf("missing buildlet config for buildlet %q", c.Name))
}
dashboard: fix docs of BuildsRepoPostSubmit, GoBootstrapURL There were some copy-paste mistakes affecting the documentation of these two identifiers. This change corrects that. Change-Id: Ic619f19c4a659a9357c816ff7fb6b0cd7b3bccd1 Reviewed-on: https://go-review.googlesource.com/c/build/+/169437 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-26 20:45:16 +03:00
// GoBootstrapURL returns the URL of a built Go 1.4+ tar.gz for the
// build configuration type c, or empty string if there isn't one.
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
func (c *BuildConfig) GoBootstrapURL(e *buildenv.Environment) string {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
return strings.Replace(c.hostConf().goBootstrapURLTmpl, "$BUCKET", e.BuildletBucket, 1)
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
}
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
// BuildletBinaryURL returns the public URL of this builder's buildlet.
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *HostConfig) BuildletBinaryURL(e *buildenv.Environment) string {
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
tmpl := c.buildletURLTmpl
all: consolidate configuration for coordinator and gomote buildenv.Environment type defines configuration options: - Coordinator uses the GCE project name to lookup config. A custom config name can be provided at runtime to override. - The conventional prod and stage project names ('symbolic-datum-552' and 'go-dashboard-dev') map to prod and staging configuration structs. - Production and staging status is explicitly defined in configuration. - GCS bucket names for buildlet, logs, and snapshots are configurable. Change-Id: I7e6d7874eb0bdfe35dbdd5fcf6212ab50d576b88 Reviewed-on: https://go-review.googlesource.com/19502 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-15 02:59:59 +03:00
return strings.Replace(tmpl, "$BUCKET", e.BuildletBucket, 1)
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
}
cmd/coordinator: shard linux race builder Fixes golang/go#11074 Change-Id: I9401122e4d7febd3e38a80be89fcb2e03969f383 Reviewed-on: https://go-review.googlesource.com/11168 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-17 18:22:14 +03:00
func (c *BuildConfig) IsRace() bool {
return strings.HasSuffix(c.Name, "-race")
}
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
func (c *BuildConfig) IsLongTest() bool {
return strings.HasSuffix(c.Name, "-longtest")
}
dashboard: set GOPROXY=off for the "go" repo to catch more network usage Updates golang/go#30612 Updates golang/go#30760 Change-Id: I296f4f7a163c1bdd59e839229c0961b21f26da89 Reviewed-on: https://go-review.googlesource.com/c/build/+/167039 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 07:44:17 +03:00
// OutboundNetworkAllowed reports whether this builder should be
// allowed to make outbound network requests. This is only enforced
// on some builders. (Currently most Linux ones)
func (c *BuildConfig) OutboundNetworkAllowed() bool {
return c.Name == "misc-vet-vetall" || c.IsLongTest()
}
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
func (c *BuildConfig) GoInstallRacePackages() []string {
if c.InstallRacePackages != nil {
return append([]string(nil), c.InstallRacePackages...)
}
if c.IsRace() {
return []string{"std"}
}
return nil
}
dashboard: move buildlet exec code from coordinator to client package Change-Id: I778ac78ed02be9f67436ec045a3816dfc24afda3 Reviewed-on: https://go-review.googlesource.com/2923 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 20:54:03 +03:00
// AllScript returns the relative path to the operating system's script to
// do the build and run its standard set of tests.
// Example values are "src/all.bash", "src/all.bat", "src/all.rc".
func (c *BuildConfig) AllScript() string {
dashboard: add sanity check Because almost all the BuildConfig methods indirectly call AllScript, add in a little defense against accident creation of BuildConfig zero values (of the value type, not nil pointers). This can happen by doing a lookup in a map[string]dashboard.BuildConfig and then calling a method on it. We should probably move everything to pointer values of BuildConfig. There aren't many value types remaining. This paranoia isn't a result of any bug yet, but rather than the result of discussion on CL 40477. Change-Id: Ic273d4de28ac8930b56bc232bf77a9ba39f99fad Reviewed-on: https://go-review.googlesource.com/40497 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 00:37:13 +03:00
if c.Name == "" {
panic("bogus BuildConfig")
}
cmd/coordinator: shard linux race builder Fixes golang/go#11074 Change-Id: I9401122e4d7febd3e38a80be89fcb2e03969f383 Reviewed-on: https://go-review.googlesource.com/11168 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-17 18:22:14 +03:00
if c.IsRace() {
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
if strings.HasPrefix(c.Name, "windows-") {
return "src/race.bat"
}
return "src/race.bash"
}
dashboard: move buildlet exec code from coordinator to client package Change-Id: I778ac78ed02be9f67436ec045a3816dfc24afda3 Reviewed-on: https://go-review.googlesource.com/2923 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 20:54:03 +03:00
if strings.HasPrefix(c.Name, "windows-") {
return "src/all.bat"
}
if strings.HasPrefix(c.Name, "plan9-") {
return "src/all.rc"
}
env: update nacl builders to be VM-based Be lazy and use sid as a base, instead of trusty. This replaces my earlier failed attempt: https://golang.org/cl/4960 Fixes golang/go#9879 Change-Id: Ia0a39c99747844b6e34bf7561c01dce49fa3b226 Reviewed-on: https://go-review.googlesource.com/5191 Reviewed-by: Russ Cox <rsc@golang.org>
2015-02-19 01:12:22 +03:00
if strings.HasPrefix(c.Name, "nacl-") {
return "src/nacltest.bash"
}
dashboard: add darwin-arm/arm64 buildlets Tested by running a local coordinator on the darwin-arm64 builder. Change-Id: Id38faaa3275d6535b897f3c65df6d01ed9c1847e Reviewed-on: https://go-review.googlesource.com/9459 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-04-29 15:54:19 +03:00
if strings.HasPrefix(c.Name, "darwin-arm") {
return "src/iostest.bash"
}
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
if strings.HasPrefix(c.Name, "misc-compile") {
dashboard: add all-compile builder Runs buildall.bash only. Change-Id: I1a68d54569ccb511d956c5b181e463503353d4d4 Reviewed-on: https://go-review.googlesource.com/9512 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2015-04-30 00:44:46 +03:00
return "src/buildall.bash"
}
dashboard: move buildlet exec code from coordinator to client package Change-Id: I778ac78ed02be9f67436ec045a3816dfc24afda3 Reviewed-on: https://go-review.googlesource.com/2923 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 20:54:03 +03:00
return "src/all.bash"
}
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
// SplitMakeRun reports whether the coordinator should first compile
// (using c.MakeScript), then snapshot, then run the tests (ideally
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// sharded) using cmd/dist test.
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
// Eventually this function should always return true (and then be deleted)
// but for now we've only set up the scripts and verified that the main
// configurations work.
func (c *BuildConfig) SplitMakeRun() bool {
switch c.AllScript() {
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
case "src/all.bash", "src/all.bat",
"src/race.bash", "src/race.bat",
"src/all.rc",
"src/nacltest.bash":
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
// These we've verified to work.
return true
}
dashboard: remove eliasnaur Android emulators Now that we hace Android emulator builders on GCE, drop the old ones running on hardware. Also update the configuration of the remaining builders now that androidtest.bash is no longer necessary. Change-Id: I09cab08b28952f967205d6c35f36eb6e05a17950 Reviewed-on: https://go-review.googlesource.com/c/164397 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 10:19:36 +03:00
// TODO(bradfitz): make iostest.bash work too. And
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
// buildall.bash should really just be N small container jobs
// instead of a "buildall.bash". Then we can delete this whole
// method.
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
return false
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
func (c *BuildConfig) IsTryOnly() bool { return c.tryOnly }
func (c *BuildConfig) NeedsGoProxy() bool { return c.needsGoProxy }
dashboard: fix docs of BuildsRepoPostSubmit, GoBootstrapURL There were some copy-paste mistakes affecting the documentation of these two identifiers. This change corrects that. Change-Id: Ic619f19c4a659a9357c816ff7fb6b0cd7b3bccd1 Reviewed-on: https://go-review.googlesource.com/c/build/+/169437 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-26 20:45:16 +03:00
// BuildsRepoPostSubmit reports whether the build configuration type c
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// should build the given repo ("go", "net", etc) and branch
// ("master", "release-branch.go1.12") as a post-submit build
// that shows up on https://build.golang.org/.
func (c *BuildConfig) BuildsRepoPostSubmit(repo, branch, goBranch string) bool {
if c.tryOnly {
dashboard: build sub-repos on more builders Change-Id: I57c12f427448281d71b8507ed950038209134143 Reviewed-on: https://go-review.googlesource.com/10915 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-11 20:01:24 +03:00
return false
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
return c.buildsRepoAtAll(repo, branch, goBranch)
cmd/coordinator: build sub-repositories Fixes golang/go#9506 Change-Id: Iebfb52386b5ba3df7d058f266e227d634d210ec6 Reviewed-on: https://go-review.googlesource.com/10732 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-05 02:45:17 +03:00
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// BuildsRepoTryBot reports whether the build configuration type c
// should build the given repo ("go", "net", etc) and branch
// ("master", "release-branch.go1.12") as a trybot.
func (c *BuildConfig) BuildsRepoTryBot(repo, branch, goBranch string) bool {
return c.tryBot != nil && c.tryBot(repo, branch, goBranch) && c.buildsRepoAtAll(repo, branch, goBranch)
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
}
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
// ShouldRunDistTest reports whether the named cmd/dist test should be
// run for this build config. The isTry parameter is whether this is
// for a trybot (pre-submit) run.
//
// In general, this returns true, unless a builder has configured it
// otherwise. Certain portable, slow tests are only run on fast builders in
// trybot mode.
func (c *BuildConfig) ShouldRunDistTest(distTest string, isTry bool) bool {
if c.shouldRunDistTest != nil {
return c.shouldRunDistTest(distTest, isTry)
}
if distTest == "api" {
// This test is slow and has the same behavior
// everywhere, so only run it on our fastest buidler
// (linux-amd64) when in trybot mode.
return !isTry || c.Name == "linux-amd64"
}
return true
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// buildsRepoAtAll reports whether we should do builds of the provided
// repo ("go", "sys", "net", etc). This applies to both post-submit
// and trybot builds. Use BuildsRepoPostSubmit for only post-submit
// or BuildsRepoTryBot for trybots.
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
//
// The branch is the branch of repo ("master",
// "release-branch.go1.12", etc); it is required. The goBranch is the
// branch of Go itself. It's required if repo != "go". When repo ==
// "go", the goBranch defaults to the value of branch.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
func (c *BuildConfig) buildsRepoAtAll(repo, branch, goBranch string) bool {
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
if goBranch == "" {
if repo == "go" {
goBranch = branch
} else {
panic("missing goBranch")
}
}
if branch == "" {
panic("missing branch")
}
if repo == "" {
panic("missing repo")
}
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
// Don't build old branches.
const minGo1x = 11
dashboard: require at least Go 1.12 for aix builder Fixes golang/go#30799 Change-Id: I0b3b53b715e2dc00f10548f74443224190003818 Reviewed-on: https://go-review.googlesource.com/c/build/+/167357 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 11:29:15 +03:00
for _, b := range []string{branch, goBranch} {
if bmaj, bmin, ok := version.ParseReleaseBranch(b); ok {
if bmaj != 1 || bmin < minGo1x {
return false
}
bmm := types.MajorMinor{bmaj, bmin}
if bmm.Less(c.MinimumGoVersion) {
return false
}
dashboard: don't test the exp repo against release branches Exp is experimental. It's hard enough keeping it passing on master. We're not going to fight release branches as well. If somebody wants to see test coverage against three versions of Go, they can graduate their code out of experimental. Change-Id: I4695ce479c47bd56a4a58dba05d367d3de020c58 Reviewed-on: https://go-review.googlesource.com/c/build/+/170518 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-03 07:45:29 +03:00
if repo == "exp" {
// Don't test exp against release branches; it's experimental.
return false
}
dashboard: require at least Go 1.12 for aix builder Fixes golang/go#30799 Change-Id: I0b3b53b715e2dc00f10548f74443224190003818 Reviewed-on: https://go-review.googlesource.com/c/build/+/167357 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 11:29:15 +03:00
}
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
}
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// Build dev.boringcrypto branches only on linux/amd64 and windows/386 (see golang.org/issue/26791).
if repo == "go" && (branch == "dev.boringcrypto" || strings.HasPrefix(branch, "dev.boringcrypto.")) {
if c.Name != "linux-amd64" && c.Name != "windows-386-2008" {
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
return false
}
}
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
if p := c.buildsRepo; p != nil {
return p(repo, branch, goBranch)
}
if repo == "go" {
return true
}
if !c.SplitMakeRun() {
return false
}
dashboard: add some "exp" trybots back, clean up some config Fixes golang/go#30781 Change-Id: I2672ca2949837c3cfe8fa6bd8011d1b9b43cca03 Reviewed-on: https://go-review.googlesource.com/c/build/+/167640 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-14 23:35:32 +03:00
switch repo {
case "go":
return true
case "term":
// no code yet in repo
return false
case "mobile":
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
// Mobile is opt-in.
return false
}
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
return true
}
cmd/coordinator, dashboard: split build into 'make' and 'run' steps Change-Id: Ib60a601ef21163d5e0d1fed3a2609d9b719c616a Reviewed-on: https://go-review.googlesource.com/10073 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-21 07:02:38 +03:00
// AllScriptArgs returns the set of arguments that should be passed to the
dashboard: add darwin-arm/arm64 buildlets Tested by running a local coordinator on the darwin-arm64 builder. Change-Id: Id38faaa3275d6535b897f3c65df6d01ed9c1847e Reviewed-on: https://go-review.googlesource.com/9459 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-04-29 15:54:19 +03:00
// all.bash-equivalent script. Usually empty.
func (c *BuildConfig) AllScriptArgs() []string {
if strings.HasPrefix(c.Name, "darwin-arm") {
return []string{"-restart"}
}
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
return append([]string(nil), c.allScriptArgs...)
dashboard: add darwin-arm/arm64 buildlets Tested by running a local coordinator on the darwin-arm64 builder. Change-Id: Id38faaa3275d6535b897f3c65df6d01ed9c1847e Reviewed-on: https://go-review.googlesource.com/9459 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-04-29 15:54:19 +03:00
}
cmd/release: add new program to build releases The release command will replace misc/makerelease from the main repo. The contents of makerelease make their way into this program. Also add GorootFinal and MakeScript methods to dashboard.BuildConfig. Change-Id: Id20f3f7a85a21f040d8cf47e438d299d57360556 Reviewed-on: https://go-review.googlesource.com/3693 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-02 15:05:01 +03:00
// MakeScript returns the relative path to the operating system's script to
// do the build.
// Example values are "src/make.bash", "src/make.bat", "src/make.rc".
func (c *BuildConfig) MakeScript() string {
if strings.HasPrefix(c.Name, "windows-") {
return "src/make.bat"
}
if strings.HasPrefix(c.Name, "plan9-") {
return "src/make.rc"
}
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
if strings.HasPrefix(c.Name, "nacl-") {
return "src/naclmake.bash"
}
cmd/release: add new program to build releases The release command will replace misc/makerelease from the main repo. The contents of makerelease make their way into this program. Also add GorootFinal and MakeScript methods to dashboard.BuildConfig. Change-Id: Id20f3f7a85a21f040d8cf47e438d299d57360556 Reviewed-on: https://go-review.googlesource.com/3693 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-02 15:05:01 +03:00
return "src/make.bash"
}
cmd/coordinator, dashboard: split build into 'make' and 'run' steps Change-Id: Ib60a601ef21163d5e0d1fed3a2609d9b719c616a Reviewed-on: https://go-review.googlesource.com/10073 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-21 07:02:38 +03:00
// MakeScriptArgs returns the set of arguments that should be passed to the
// make.bash-equivalent script. Usually empty.
func (c *BuildConfig) MakeScriptArgs() []string {
return c.AllScriptArgs()
}
cmd/release: add new program to build releases The release command will replace misc/makerelease from the main repo. The contents of makerelease make their way into this program. Also add GorootFinal and MakeScript methods to dashboard.BuildConfig. Change-Id: Id20f3f7a85a21f040d8cf47e438d299d57360556 Reviewed-on: https://go-review.googlesource.com/3693 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-02 15:05:01 +03:00
// GorootFinal returns the default install location for
// releases for this platform.
func (c *BuildConfig) GorootFinal() string {
if strings.HasPrefix(c.Name, "windows-") {
return "c:\\go"
}
return "/usr/local/go"
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
// MachineType returns the GCE machine type to use for this builder.
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *HostConfig) MachineType() string {
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
if v := c.machineType; v != "" {
return v
}
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
if c.IsContainer() {
// Set a higher default machine size for containers,
// so their /workdir tmpfs can be larger. The COS
// image has no swap, so we want to make sure the
// /workdir fits completely in memory.
return "n1-standard-4" // 4 CPUs, 15GB RAM
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
return "n1-highcpu-2"
}
cmd/coordinator: generate table of builders To replace the always-out-of-date wiki. Change-Id: I63d56d4bcf9252e56c833eaa15656b0321ac91dc Reviewed-on: https://go-review.googlesource.com/9571 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-01 03:03:01 +03:00
// ShortOwner returns a short human-readable owner.
func (c BuildConfig) ShortOwner() string {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
owner := c.hostConf().Owner
if owner == "" {
cmd/coordinator: generate table of builders To replace the always-out-of-date wiki. Change-Id: I63d56d4bcf9252e56c833eaa15656b0321ac91dc Reviewed-on: https://go-review.googlesource.com/9571 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-01 03:03:01 +03:00
return "go-dev"
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
return strings.TrimSuffix(owner, "@golang.org")
cmd/coordinator: generate table of builders To replace the always-out-of-date wiki. Change-Id: I63d56d4bcf9252e56c833eaa15656b0321ac91dc Reviewed-on: https://go-review.googlesource.com/9571 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-01 03:03:01 +03:00
}
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
// OwnerGithub returns the Github handle of the owner.
func (c BuildConfig) OwnerGithub() string {
return c.hostConf().OwnerGithub
}
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
// PoolName returns a short summary of the builder's host type for the
all: upgrade farmer.golang.org links to https Since farmer.golang.org has a valid certificate in production now, we should use that certificate for retrieving data. Change-Id: I3685d84953ea11e6ac09b4692fa27c73538a565c Reviewed-on: https://go-review.googlesource.com/39290 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-03 17:51:28 +03:00
// https://farmer.golang.org/builders page.
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
func (c *HostConfig) PoolName() string {
switch {
case c.IsReverse:
return "Reverse (dedicated machine/VM)"
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsVM():
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
return "GCE VM"
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsContainer():
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
return "Container"
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
}
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
panic("unknown builder type")
}
env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx This adds scripts to create a new builder host image that acts like Container-Optimized OS (has docker, runs konlet on startup) but with a Debian 9 kernel + userspace that permits KVM for nested virtualization. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884 Reviewed-on: https://go-review.googlesource.com/c/163301 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-22 00:10:54 +03:00
// ContainerVMImage returns the base VM name (not the fully qualified
// URL resource name of the VM) that starts the konlet program that
// pulls & runs a container. This method is only applicable when
// c.IsContainer() is true.
func (c *HostConfig) ContainerVMImage() string {
if c.KonletVMImage != "" {
return c.KonletVMImage
}
return "debian-stretch-vmx"
}
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
// IsHermetic reports whether this host config gets a fresh
// environment (including /usr, /var, etc) for each execution. This is
// true for VMs, GKE, and reverse buildlets running their containers
// running in Docker, but false on some reverse buildlets.
func (c *HostConfig) IsHermetic() bool {
switch {
case c.IsReverse:
return c.HermeticReverse
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsVM():
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
return true
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsContainer():
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
return true
}
panic("unknown builder type")
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
}
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
// GCENumCPU reports the number of GCE CPUs this buildlet requires.
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *HostConfig) GCENumCPU() int {
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
t := c.MachineType()
n, _ := strconv.Atoi(t[strings.LastIndex(t, "-")+1:])
return n
}
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
func (c *BuildConfig) NumTestHelpers(isTry bool) int {
if isTry && c.numTryTestHelpers != 0 {
return c.numTryTestHelpers
}
return c.numTestHelpers
}
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
// defaultTrySet returns a trybot policy function that reports whether
// a project should use trybots. All the default projects are included,
// plus any given in extraProj.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
func defaultTrySet(extraProj ...string) func(proj, branch, goBranch string) bool {
return func(proj, branch, goBranch string) bool {
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
if proj == "go" {
return true
}
for _, p := range extraProj {
if proj == p {
return true
}
}
switch proj {
dashboard: add some "exp" trybots back, clean up some config Fixes golang/go#30781 Change-Id: I2672ca2949837c3cfe8fa6bd8011d1b9b43cca03 Reviewed-on: https://go-review.googlesource.com/c/build/+/167640 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-14 23:35:32 +03:00
case "grpc-review":
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
return false
}
return true
}
}
// explicitTrySet returns a trybot policy function that reports
// whether a project should use trybots. Only the provided projects in
// projs are enabled.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
func explicitTrySet(projs ...string) func(proj, branch, goBranch string) bool {
return func(proj, branch, goBranch string) bool {
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
for _, p := range projs {
if proj == p {
return true
}
}
return false
}
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
func init() {
dashboard, env: add FreeBSD 9.3 support. Differences from 10.1 support: - Uses gcc in base image, 10.1 uses clang. - Uses dig in startup diagnostic, 10.1 uses drill. Fixes #8639 Change-Id: I298bdf7d76c91395ae56606d1e2f0176eb0eb792 Reviewed-on: https://go-review.googlesource.com/3173 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-22 08:16:54 +03:00
addBuilder(BuildConfig{
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
Name: "freebsd-amd64-gce93",
HostType: "host-freebsd-93-gce",
buildsRepo: disabledBuilder,
MaxAtOnce: 2,
dashboard, env: add FreeBSD 9.3 support. Differences from 10.1 support: - Uses gcc in base image, 10.1 uses clang. - Uses dig in startup diagnostic, 10.1 uses drill. Fixes #8639 Change-Id: I298bdf7d76c91395ae56606d1e2f0176eb0eb792 Reviewed-on: https://go-review.googlesource.com/3173 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-22 08:16:54 +03:00
})
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
addBuilder(BuildConfig{
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
Name: "freebsd-amd64-10_3",
HostType: "host-freebsd-10_3",
buildsRepo: func(repo, branch, goBranch string) bool {
return branch == "release-branch.go1.11" || goBranch == "release-branch.go1.12"
},
tryBot: func(repo, branch, goBranch string) bool {
return branch == "release-branch.go1.11" || branch == "release-branch.go1.12"
},
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
})
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
addBuilder(BuildConfig{
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
Name: "freebsd-amd64-10_4",
HostType: "host-freebsd-10_4",
buildsRepo: func(repo, branch, goBranch string) bool {
return goBranch == "release-branch.go1.11" || goBranch == "release-branch.go1.12"
},
tryBot: nil,
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
MaxAtOnce: 2,
})
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
addBuilder(BuildConfig{
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
Name: "freebsd-amd64-11_1",
HostType: "host-freebsd-11_1",
tryBot: nil,
buildsRepo: func(repo, branch, goBranch string) bool {
return goBranch == "release-branch.go1.11" || goBranch == "release-branch.go1.12"
},
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: fasterTrybots,
dashboard: update Windows builders, remove old manual Windows builder And clean up some other builders. Change-Id: I1f71836da7008e58d3e76e2cc3170e96cd57ddf6 Reviewed-on: https://go-review.googlesource.com/51432 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-27 07:52:12 +03:00
numTryTestHelpers: 4,
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
})
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
addBuilder(BuildConfig{
Name: "freebsd-amd64-11_2",
HostType: "host-freebsd-11_2",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("sys"),
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: fasterTrybots,
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
numTryTestHelpers: 4,
MaxAtOnce: 2,
})
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
addBuilder(BuildConfig{
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
Name: "freebsd-amd64-12_0",
HostType: "host-freebsd-12_0",
MinimumGoVersion: types.MajorMinor{1, 11},
tryBot: defaultTrySet("sys"),
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: fasterTrybots,
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
numTryTestHelpers: 4,
MaxAtOnce: 2,
})
dashboard: add freebsd-386-12_0 Fixes golang/go#30787 Change-Id: I33d892fe56abcb797c76541fb3b3a12ffe69c598 Reviewed-on: https://go-review.googlesource.com/c/build/+/167199 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-13 00:13:50 +03:00
addBuilder(BuildConfig{
Name: "freebsd-386-12_0",
HostType: "host-freebsd-12_0",
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: fasterTrybots,
dashboard: add freebsd-386-12_0 Fixes golang/go#30787 Change-Id: I33d892fe56abcb797c76541fb3b3a12ffe69c598 Reviewed-on: https://go-review.googlesource.com/c/build/+/167199 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-13 00:13:50 +03:00
numTryTestHelpers: 4,
MaxAtOnce: 2,
})
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
addBuilder(BuildConfig{
dashboard: reenable FreeBSD 9.3/10.1, OpenBSD, and Plan9 builders The GCE bug is fixed and these VMs boot again. (Previously: golang.org/cl/50011) Change-Id: I3f330b8d0c6a6ae6708b2299bba13945a06fa36f Reviewed-on: https://go-review.googlesource.com/53270 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-04 07:25:07 +03:00
Name: "freebsd-amd64-race",
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
HostType: "host-freebsd-11_1",
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
})
addBuilder(BuildConfig{
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
Name: "freebsd-386-10_3",
HostType: "host-freebsd-10_3",
buildsRepo: func(repo, branch, goBranch string) bool {
return goBranch == "release-branch.go1.11" || goBranch == "release-branch.go1.12"
},
dashboard: reenable FreeBSD 9.3/10.1, OpenBSD, and Plan9 builders The GCE bug is fixed and these VMs boot again. (Previously: golang.org/cl/50011) Change-Id: I3f330b8d0c6a6ae6708b2299bba13945a06fa36f Reviewed-on: https://go-review.googlesource.com/53270 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-04 07:25:07 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
addBuilder(BuildConfig{
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
Name: "freebsd-386-10_4",
HostType: "host-freebsd-10_4",
buildsRepo: func(repo, branch, goBranch string) bool {
return goBranch == "release-branch.go1.11" || goBranch == "release-branch.go1.12"
},
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
MaxAtOnce: 2,
})
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "freebsd-386-11_1",
HostType: "host-freebsd-11_1",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
return goBranch == "release-branch.go1.11" || goBranch == "release-branch.go1.12"
},
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
MaxAtOnce: 2,
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
})
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
addBuilder(BuildConfig{
Name: "freebsd-386-11_2",
HostType: "host-freebsd-11_2",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
tryBot: explicitTrySet("sys"),
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
MaxAtOnce: 2,
})
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "linux-386",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: fasterTrybots,
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
env, buildlet, dashboard: support outbound firewalls for Linux Updates golang/go#30612 Change-Id: Ib13a286d0944a7f4a13b9e93a01533693052858b Reviewed-on: https://go-review.googlesource.com/c/build/+/165637 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 05:06:17 +03:00
env: []string{
"GOARCH=386",
"GOHOSTARCH=386",
"GO_DISABLE_OUTBOUND_NETWORK=1",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
numTestHelpers: 1,
numTryTestHelpers: 3,
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
addBuilder(BuildConfig{
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
Name: "linux-386-387",
Notes: "GO386=387",
buildsRepo: func(repo, branch, goBranch string) bool {
return repo == "go" || (repo == "crypto" && branch == "master" && goBranch == "master")
},
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386", "GO386=387"},
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
addBuilder(BuildConfig{
env, buildlet, dashboard: support outbound firewalls for Linux Updates golang/go#30612 Change-Id: Ib13a286d0944a7f4a13b9e93a01533693052858b Reviewed-on: https://go-review.googlesource.com/c/build/+/165637 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 05:06:17 +03:00
Name: "linux-amd64",
env/linux-x86-stretch, dashboard: add swig, move linux-amd64 from jessie Fixes golang/go#28857 Change-Id: Ifdd62dc8eac6ce52f9dc9ce8b3d2fcd92d4d1b26 Reviewed-on: https://go-review.googlesource.com/c/build/+/170282 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2019-04-02 02:16:17 +03:00
HostType: "host-linux-stretch",
env, buildlet, dashboard: support outbound firewalls for Linux Updates golang/go#30612 Change-Id: Ib13a286d0944a7f4a13b9e93a01533693052858b Reviewed-on: https://go-review.googlesource.com/c/build/+/165637 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 05:06:17 +03:00
tryBot: defaultTrySet(),
env: []string{
"GO_DISABLE_OUTBOUND_NETWORK=1",
},
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 3,
numTestHelpers: 1,
numTryTestHelpers: 4,
RunBench: true,
all: run dev coordinator locally, allow debug in prod * flag allows debug (trigger a build) in prod mode * use default zone (us-central1-f) in staging * compute.CloudPlatformScope satisfies compute and storage scope requirements * new buildlet for linux-amd64 on Kubernetes Updates golang/go#12546 Change-Id: I1a0cbc0e0c0552b7ac56943c646378194508d48f Reviewed-on: https://go-review.googlesource.com/17102 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-11-20 21:00:06 +03:00
})
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
addBuilder(BuildConfig{
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
Name: "linux-amd64-vmx",
HostType: "host-linux-stretch-vmx",
MaxAtOnce: 1,
buildsRepo: disabledBuilder,
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
})
dashboard: remove alpine builder It's doing no good being red all the time, especially as nobody is working on it. Updates golang/go#22689 Updates golang/go#19938 Change-Id: Icb947878d85e920f24ea458eb0f319844ca5bd60 Reviewed-on: https://go-review.googlesource.com/78575 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-18 06:31:35 +03:00
const testAlpine = false // Issue 22689 (hide all red builders), Issue 19938 (get Alpine passing)
if testAlpine {
addBuilder(BuildConfig{
Name: "linux-amd64-alpine",
HostType: "host-linux-x86-alpine",
})
}
dashboard, coordinator: add vetall builder Change-Id: I331a27050e6f068aa47afa9012a9bb50e3b0c46a Reviewed-on: https://go-review.googlesource.com/37573 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-01-09 23:02:06 +03:00
// Add the -vetall builder. The builder name suffix "-vetall" is recognized by cmd/dist/test.go
// to only run the "go vet std cmd" test and no others.
addBuilder(BuildConfig{
dashboard: only run misc-vet-vetall for master As of CL 169198 for golang/go#31040, the coordinator no longer supports placing the x/tools files out for Go 1.11 and Go 1.12's cmd/vet/all to find, so their misc-vet-vetall builds will start failing. We could in theory still either backport CL 169237 to release-branch.go1.11 and release-branch.go1.12, and/or we could keep running the misc-vet-vetall builder for the future Go 1.13 release branch, but it's not obviously even worth it; the vetall builder is most useful during development anyway. The tree is pretty stable once it's release branch time. The main use of misc-vet-vetall is catching assembly vet failures (since go test now does most the other vet checks), and assembly doesn't often change in release branches. Updates golang/go#31040 Change-Id: I7b827ecbcd206f3dcf63e04cc94fb78854befd7d Reviewed-on: https://go-review.googlesource.com/c/build/+/169339 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-26 19:33:00 +03:00
Name: "misc-vet-vetall",
HostType: "host-linux-jessie",
Notes: "Runs vet over the standard library.",
buildsRepo: func(repo, branch, goBranch string) bool {
return repo == "go" && branch == "master"
},
tryBot: defaultTrySet(),
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
"GO_DISABLE_OUTBOUND_NETWORK=1",
},
dashboard, coordinator: add vetall builder Change-Id: I331a27050e6f068aa47afa9012a9bb50e3b0c46a Reviewed-on: https://go-review.googlesource.com/37573 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-01-09 23:02:06 +03:00
numTestHelpers: 5,
})
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
addMiscCompile := func(suffix, rx string) {
addBuilder(BuildConfig{
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
Name: "misc-compile" + suffix,
HostType: "host-linux-jessie",
tryBot: defaultTrySet(),
env: []string{
"GO_DISABLE_OUTBOUND_NETWORK=1",
},
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
tryOnly: true,
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
CompileOnly: true,
Notes: "Runs buildall.sh to cross-compile std packages for " + rx + ", but doesn't run any tests.",
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
allScriptArgs: []string{
// Filtering pattern to buildall.bash:
rx,
},
})
}
dashboard: add more misc-compile trybots s390x notably was missing. Also add the mobile ones. Change-Id: Id6be52b73fac897980d4e94ac5c5b07027f5b4e7 Reviewed-on: https://go-review.googlesource.com/114315 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-24 01:19:10 +03:00
addMiscCompile("", "^(linux-arm64|linux-s390x|solaris-amd64|darwin-386)$") // 4 ports
addMiscCompile("-nacl", "^nacl") // 3
addMiscCompile("-mips", "^linux-mips") // 4
addMiscCompile("-ppc", "^linux-ppc64") // 2
addMiscCompile("-plan9", "^plan9-") // 3
addMiscCompile("-freebsd", "^freebsd-") // 3
addMiscCompile("-netbsd", "^netbsd-") // 3
addMiscCompile("-openbsd", "^openbsd-") // 3
dashboard: remove mobile misc-compile trybots I guess they never worked. Updates golang/go#25963 Change-Id: I1409a704670032410c3c6b87cbbb876fc96dfb22 Reviewed-on: https://go-review.googlesource.com/119856 Reviewed-by: Filippo Valsorda <filippo@golang.org>
2018-06-19 23:50:06 +03:00
// TODO: Issue 25963, get the misc-compile trybots for mobile working, and then:
// addMiscCompile("-mobile", "(^android|darwin-arm64)") // 5 ports
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-amd64-nocgo",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 1,
Notes: "cgo disabled",
dashboard: add some "exp" trybots back, clean up some config Fixes golang/go#30781 Change-Id: I2672ca2949837c3cfe8fa6bd8011d1b9b43cca03 Reviewed-on: https://go-review.googlesource.com/c/build/+/167640 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-14 23:35:32 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
switch repo {
case "perf":
// Requires sqlite, which requires cgo.
return false
case "mobile":
return false
}
return true
},
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
env: []string{
"CGO_ENABLED=0",
env, buildlet, dashboard: support outbound firewalls for Linux Updates golang/go#30612 Change-Id: Ib13a286d0944a7f4a13b9e93a01533693052858b Reviewed-on: https://go-review.googlesource.com/c/build/+/165637 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 05:06:17 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
// This USER=root was required for Docker-based builds but probably isn't required
// in the VM anymore, since the buildlet probably already has this in its environment.
// (It was required because without cgo, it couldn't find the username)
"USER=root",
},
})
addBuilder(BuildConfig{
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
Name: "linux-amd64-noopt",
Notes: "optimizations and inlining disabled",
HostType: "host-linux-jessie",
buildsRepo: onlyGo,
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
"GO_DISABLE_OUTBOUND_NETWORK=1",
"GO_GCFLAGS=-N -l",
},
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 1,
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
dashboard: add SSA check builder Now that SSA checks are off by default, let's run a builder with the checks on just to make sure they keep passing. Change-Id: I7edd2a2865681a135ccf755be7a14f8645c0a8aa Reviewed-on: https://go-review.googlesource.com/20882 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-19 01:30:38 +03:00
addBuilder(BuildConfig{
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
Name: "linux-amd64-ssacheck",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 1,
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
buildsRepo: onlyGo,
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil, // TODO: add a func to conditionally run this trybot if compiler dirs are touched
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
CompileOnly: true,
Notes: "SSA internal checks enabled",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
"GO_DISABLE_OUTBOUND_NETWORK=1",
"GO_GCFLAGS=-d=ssa/check/on,dclstack",
},
cmd/coordinator: move git ancestor checking before buildlet creation, generalize This also adds GoDeps to the builders config file and adds a dependency for the ssacheck builder, which currently fails on older release branches with: > Failed on linux-amd64-ssacheck: > https://storage.googleapis.com/go-build-log/1255edaf/linux-amd64-ssacheck_18f7e015.log > > ... > compile: unknown debug key -d dclstack Change-Id: I4ad0a298705feed56f467f3c8d9488f8be2e4af5 Reviewed-on: https://go-review.googlesource.com/43774 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-20 00:23:14 +03:00
GoDeps: []string{
"f65abf6ddc8d1f3d403a9195fd74eaffa022b07f", // adds dclstack
},
dashboard: add SSA check builder Now that SSA checks are off by default, let's run a builder with the checks on just to make sure they keep passing. Change-Id: I7edd2a2865681a135ccf755be7a14f8645c0a8aa Reviewed-on: https://go-review.googlesource.com/20882 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-19 01:30:38 +03:00
})
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-racecompile",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil, // TODO: add a func to conditionally run this trybot if compiler dirs are touched
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 1,
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
CompileOnly: true,
cmd/coordinator: make linux-amd64-racecompile rebuild std/cmd with -c=128 Updates golang/go#19962 Change-Id: I8b938a39b33d4e6b9b8dc5bc0802da95e889ee46 Reviewed-on: https://go-review.googlesource.com/41613 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-25 00:14:05 +03:00
SkipSnapshot: true,
StopAfterMake: true,
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
InstallRacePackages: []string{"cmd/compile"},
Notes: "race-enabled cmd/compile",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
"GO_DISABLE_OUTBOUND_NETWORK=1",
},
cmd/coordinator: move git ancestor checking before buildlet creation, generalize This also adds GoDeps to the builders config file and adds a dependency for the ssacheck builder, which currently fails on older release branches with: > Failed on linux-amd64-ssacheck: > https://storage.googleapis.com/go-build-log/1255edaf/linux-amd64-ssacheck_18f7e015.log > > ... > compile: unknown debug key -d dclstack Change-Id: I4ad0a298705feed56f467f3c8d9488f8be2e4af5 Reviewed-on: https://go-review.googlesource.com/43774 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-20 00:23:14 +03:00
GoDeps: []string{
"22f1b56dab29d397d2bdbdd603d85e60fb678089", // adds cmd/compile -c; Issue 20222
},
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
})
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
Name: "linux-amd64-race",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 1,
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: fasterTrybots,
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
numTestHelpers: 1,
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
numTryTestHelpers: 5,
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
"GO_DISABLE_OUTBOUND_NETWORK=1",
},
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
})
env: convert linux-x86-clang to be VM-based Change-Id: I583d751652dec9176169b4d4705f3813506017e3 Reviewed-on: https://go-review.googlesource.com/4790 Reviewed-by: Minux Ma <minux@golang.org>
2015-02-13 06:45:02 +03:00
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-386-clang",
HostType: "host-linux-clang",
MaxAtOnce: 1,
Notes: "Debian jessie + clang 3.9 instead of gcc",
env: []string{"CC=/usr/bin/clang", "GOHOSTARCH=386"},
env: convert linux-x86-clang to be VM-based Change-Id: I583d751652dec9176169b4d4705f3813506017e3 Reviewed-on: https://go-review.googlesource.com/4790 Reviewed-by: Minux Ma <minux@golang.org>
2015-02-13 06:45:02 +03:00
})
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-amd64-clang",
HostType: "host-linux-clang",
MaxAtOnce: 1,
Notes: "Debian jessie + clang 3.9 instead of gcc",
env: []string{"CC=/usr/bin/clang"},
env: convert linux-x86-clang to be VM-based Change-Id: I583d751652dec9176169b4d4705f3813506017e3 Reviewed-on: https://go-review.googlesource.com/4790 Reviewed-by: Minux Ma <minux@golang.org>
2015-02-13 06:45:02 +03:00
})
env: update linux-x86-sid to be a VM Fixes #9873 Change-Id: I08c501ef52ebde4cdbc48fe45a5342488000c433 Reviewed-on: https://go-review.googlesource.com/4874 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-14 06:01:32 +03:00
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-386-sid",
HostType: "host-linux-sid",
Notes: "Debian sid (unstable)",
MaxAtOnce: 1,
env: []string{"GOHOSTARCH=386"},
env: update linux-x86-sid to be a VM Fixes #9873 Change-Id: I08c501ef52ebde4cdbc48fe45a5342488000c433 Reviewed-on: https://go-review.googlesource.com/4874 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-14 06:01:32 +03:00
})
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-amd64-sid",
HostType: "host-linux-sid",
MaxAtOnce: 1,
Notes: "Debian sid (unstable)",
env: update linux-x86-sid to be a VM Fixes #9873 Change-Id: I08c501ef52ebde4cdbc48fe45a5342488000c433 Reviewed-on: https://go-review.googlesource.com/4874 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-14 06:01:32 +03:00
})
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-androidemu",
HostType: "host-android-amd64-emu",
env: []string{
"GOARCH=amd64",
"GOOS=linux",
"CGO_ENABLED=1",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
tryBot: func(repo, branch, goBranch string) bool {
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
// Only for mobile repo for now, not "go":
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
return repo == "mobile" && branch == "master" && goBranch == "master"
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
return repo == "mobile" && branch == "master" && goBranch == "master"
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
Notes: "Runs GOOS=linux but with the Android emulator attached, for running x/mobile host tests.",
})
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-stretch",
HostType: "host-linux-stretch",
MaxAtOnce: 1,
Notes: "Debian Stretch",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
"GO_DISABLE_OUTBOUND_NETWORK=1",
},
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
})
dashboard: add "longtest" builder to run all tests, without -short mode Fixes golang/go#12508 Change-Id: I62e1e296f10c7622c1967a5530c4b0e07c136dfd Reviewed-on: https://go-review.googlesource.com/113436 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-16 22:50:39 +03:00
addBuilder(BuildConfig{
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
Name: "linux-amd64-longtest",
dashboard: increase timeout scale for longtest builders, add more CPU Updates golang/go#25886 Change-Id: I5168e291ab77cbd3843bdc39e319a68dfa65aedd Reviewed-on: https://go-review.googlesource.com/c/build/+/167638 Reviewed-by: Russ Cox <rsc@golang.org>
2019-03-14 19:06:57 +03:00
HostType: "host-linux-stretch-morecpu",
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
MaxAtOnce: 1,
Notes: "Debian Stretch with go test -short=false",
buildsRepo: func(repo, branch, goBranch string) bool {
return repo == "go" || (branch == "master" && goBranch == "master")
},
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
needsGoProxy: true, // for cmd/go module tests
dashboard: increase TIMEOUT_SCALE to 2 for linux-amd64-longtest runtime tests take about 180s in long mode, too close to the timeout limit (180 seconds), and this is causing sporadic timeout failures like https://build.golang.org/log/cda568a39f396e3fee64778a475528be452632ed Bump up the builder's timeout by 2x. Fixes golang/go#25629 Change-Id: I77e0f86f0b236cb953457ebdcbfc5763c2c6d54b Reviewed-on: https://go-review.googlesource.com/115016 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-29 22:03:23 +03:00
env: []string{
"GO_TEST_SHORT=0",
dashboard: increase timeout scale for longtest builders, add more CPU Updates golang/go#25886 Change-Id: I5168e291ab77cbd3843bdc39e319a68dfa65aedd Reviewed-on: https://go-review.googlesource.com/c/build/+/167638 Reviewed-by: Russ Cox <rsc@golang.org>
2019-03-14 19:06:57 +03:00
"GO_TEST_TIMEOUT_SCALE=5", // give them lots of time
dashboard: increase TIMEOUT_SCALE to 2 for linux-amd64-longtest runtime tests take about 180s in long mode, too close to the timeout limit (180 seconds), and this is causing sporadic timeout failures like https://build.golang.org/log/cda568a39f396e3fee64778a475528be452632ed Bump up the builder's timeout by 2x. Fixes golang/go#25629 Change-Id: I77e0f86f0b236cb953457ebdcbfc5763c2c6d54b Reviewed-on: https://go-review.googlesource.com/115016 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-29 22:03:23 +03:00
},
dashboard: add "longtest" builder to run all tests, without -short mode Fixes golang/go#12508 Change-Id: I62e1e296f10c7622c1967a5530c4b0e07c136dfd Reviewed-on: https://go-review.googlesource.com/113436 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-16 22:50:39 +03:00
})
dashboard: add linux-arm-qemu builder (disabled) and adjust plan9 builder Change-Id: If2b4074f9bc1b44e97897096eb173b42c8323250 Reviewed-on: https://go-review.googlesource.com/6411 Reviewed-by: Salman Aljammaz <salman.aljammaz@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-03-01 20:23:57 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
Name: "linux-arm",
all: dockerize scaleway builders Fixes golang/go#20397 Change-Id: I353e879e61aeef56f0c228bbd37cc1eea2ce8b4e Reviewed-on: https://go-review.googlesource.com/43612 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-18 00:28:46 +03:00
HostType: "host-linux-arm-scaleway",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil, // Issue 22748, Issue 22749
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
FlakyNet: true,
numTestHelpers: 2,
numTryTestHelpers: 7,
dashboard, cmd/coordinator: sharded ARM trybot-only builders This is a quick & hacky version of our eventual sharding plan. Later we want to run make.bash once, snapshot it, and then intelligently schedule all the tests out over N machines (starting with the built snapshot), giving each the next test to run as they become idle. And then we want to stitch together all of their output. But this CL just makes 6 ARM builders, each of which runs make.bash (4 minutes) and then each which runs a statically-configured subset of the tests. Based on measurements, each should run between 8.2 and 10.6 minutes. We can break up some of the larger test cases later into more shardable pieces (in particular: "test"). Update golang/go#10029 Change-Id: Ib3e008b61ce6f28d799ae298c8467bf62d89e5d4 Reviewed-on: https://go-review.googlesource.com/10056 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-14 23:39:58 +03:00
})
dashboard: add linux-arm and linux-arm-arm5 builders Change-Id: Ib8997f76bdf7965d34543cbb6221d6bdb25246fa Reviewed-on: https://go-review.googlesource.com/9840 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2015-05-07 21:36:25 +03:00
addBuilder(BuildConfig{
dashboard: add a make.bash-only builder on real ARM hardware This is a new builder in prep for the change to the "linux-arm" builder where the GOARCH=arm make.bash will be cross-compiled from a Kubernetes container on fast hardware. Updates golang/go#17105 (cross-compile ARM builders' make.bash) Updates golang/go#17104 (5 minute trybots) Change-Id: Icfd2644d77639f731151abe54839322960418254 Reviewed-on: https://go-review.googlesource.com/29670 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 02:29:31 +03:00
Name: "linux-arm-nativemake",
Notes: "runs make.bash on real ARM hardware, but does not run tests",
dashboard: fix build Tests (and binaries) panicked before this. Change-Id: If984e9e30016d403af2a94971fc4a419a269b73a Reviewed-on: https://go-review.googlesource.com/46510 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-23 17:02:38 +03:00
HostType: "host-linux-arm-scaleway",
dashboard: add a make.bash-only builder on real ARM hardware This is a new builder in prep for the change to the "linux-arm" builder where the GOARCH=arm make.bash will be cross-compiled from a Kubernetes container on fast hardware. Updates golang/go#17105 (cross-compile ARM builders' make.bash) Updates golang/go#17104 (5 minute trybots) Change-Id: Icfd2644d77639f731151abe54839322960418254 Reviewed-on: https://go-review.googlesource.com/29670 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 02:29:31 +03:00
StopAfterMake: true,
})
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
addBuilder(BuildConfig{
Name: "linux-arm-arm5spacemonkey",
HostType: "host-linux-arm5spacemonkey",
dashboard: increase arm5 builder timeouts Change-Id: I1c1af4f5f6d98ca78da2ab27c3006da867492777 Reviewed-on: https://go-review.googlesource.com/82935 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-12-08 20:39:57 +03:00
env: []string{
"GOARM=5",
"GO_TEST_TIMEOUT_SCALE=4", // arm is normally 2; double that.
},
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
return branch == "master" && goBranch == "master"
},
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: func(distTest string, isTry bool) bool {
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
if strings.Contains(distTest, "vendor/github.com/google/pprof") {
// Not worth it. And broken.
return false
}
dashboard: skip some slow/broken tests on arm5 builder Updates golang/go#24754 Change-Id: Ie5e94a8fd0b76e6394249a734d8fc1a61650dbad Reviewed-on: https://go-review.googlesource.com/106118 Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2018-04-10 23:03:26 +03:00
if distTest == "api" {
// Broken on this build config (Issue
// 24754), and not worth it on slow
// builder. It's covered by other
// builders anyway.
return false
}
if strings.HasPrefix(distTest, "test:") {
// Slow, and not worth it on slow builder.
return false
}
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
return true
},
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
})
env: update nacl builders to be VM-based Be lazy and use sid as a base, instead of trusty. This replaces my earlier failed attempt: https://golang.org/cl/4960 Fixes golang/go#9879 Change-Id: Ia0a39c99747844b6e34bf7561c01dce49fa3b226 Reviewed-on: https://go-review.googlesource.com/5191 Reviewed-by: Russ Cox <rsc@golang.org>
2015-02-19 01:12:22 +03:00
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "nacl-386",
HostType: "host-nacl-kube",
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
buildsRepo: onlyGo,
dashboard: revert accidental inclusion of nacl subrepo trybots Fixes golang/go#28582 Change-Id: Ie3bbb9f4dc1a12c5c4a838165c81f5e2a762a9fe Reviewed-on: https://go-review.googlesource.com/c/147337 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-11-04 04:44:35 +03:00
tryBot: explicitTrySet("go"),
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 2,
numTryTestHelpers: 3,
env: []string{"GOOS=nacl", "GOARCH=386", "GOHOSTOS=linux", "GOHOSTARCH=amd64"},
all: upgrade nacl builders to pepper_44 and Kube Updates golang/go#12379 Change-Id: I40bd6e37953bc509f557ff295374b51de550726c Reviewed-on: https://go-review.googlesource.com/18946 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-01-27 02:13:47 +03:00
})
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "nacl-amd64p32",
HostType: "host-nacl-kube",
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
buildsRepo: onlyGo,
dashboard: revert accidental inclusion of nacl subrepo trybots Fixes golang/go#28582 Change-Id: Ie3bbb9f4dc1a12c5c4a838165c81f5e2a762a9fe Reviewed-on: https://go-review.googlesource.com/c/147337 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-11-04 04:44:35 +03:00
tryBot: explicitTrySet("go"),
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 2,
numTryTestHelpers: 3,
env: []string{"GOOS=nacl", "GOARCH=amd64p32", "GOHOSTOS=linux", "GOHOSTARCH=amd64"},
all: upgrade nacl builders to pepper_44 and Kube Updates golang/go#12379 Change-Id: I40bd6e37953bc509f557ff295374b51de550726c Reviewed-on: https://go-review.googlesource.com/18946 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-01-27 02:13:47 +03:00
})
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
addBuilder(BuildConfig{
dashboard: speed up js/wasm trybots We don't need to run all the tests in a trybot. Run just enough to get good coverage, without going over 5 minutes. Any regressions elsewhere will be caught by the build.golang.org (slower) runs. Change-Id: I32f1fc17681bfb509844d2bd35b05c950806d283 Reviewed-on: https://go-review.googlesource.com/121938 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-28 23:30:35 +03:00
Name: "js-wasm",
HostType: "host-js-wasm",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("go"),
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
switch repo {
case "go":
return true
case "mobile", "benchmarks", "debug", "perf", "talks", "tools", "tour", "website":
return false
default:
return branch == "master" && goBranch == "master"
}
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
},
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: func(distTest string, isTry bool) bool {
dashboard: speed up js/wasm trybots We don't need to run all the tests in a trybot. Run just enough to get good coverage, without going over 5 minutes. Any regressions elsewhere will be caught by the build.golang.org (slower) runs. Change-Id: I32f1fc17681bfb509844d2bd35b05c950806d283 Reviewed-on: https://go-review.googlesource.com/121938 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-28 23:30:35 +03:00
if isTry {
if strings.HasPrefix(distTest, "test:") {
return false
}
if strings.Contains(distTest, "/internal/") ||
strings.Contains(distTest, "vendor/golang.org/x/arch") {
return false
}
switch distTest {
case "cmd/go", "nolibgcc:crypto/x509":
return false
}
return true
}
return true
},
dashboard: enable trybots for js/wasm Fixes golang/go#26015 Change-Id: I25656a8e2918196cf950e29184c783503da9f905 Reviewed-on: https://go-review.googlesource.com/120775 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-06-25 21:12:11 +03:00
numTryTestHelpers: 4,
GoDeps: []string{
"3dced519cbabc213df369d9112206986e62687fa", // first passing commit
},
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
env: []string{
"GOOS=js", "GOARCH=wasm", "GOHOSTOS=linux", "GOHOSTARCH=amd64",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/workdir/go/misc/wasm",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
},
})
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
addBuilder(BuildConfig{
dashboard: switch to OpenBSD 6.0 completely, remove OpenBSD 5.8 Updates golang/go#16951 Change-Id: I7b92363671e0ba4c6ab80cef60e7c8600687f440 Reviewed-on: https://go-review.googlesource.com/34384 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-12-15 01:41:25 +03:00
Name: "openbsd-amd64-60",
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
HostType: "host-openbsd-amd64-60",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
buildsRepo: disabledBuilder,
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 1,
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
numTestHelpers: 2,
numTryTestHelpers: 5,
})
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "openbsd-386-60",
HostType: "host-openbsd-386-60",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
buildsRepo: disabledBuilder,
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
MaxAtOnce: 1,
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
env: []string{
// cmd/go takes ~192 seconds on openbsd-386
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
})
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "openbsd-386-62",
HostType: "host-openbsd-386-62",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
MaxAtOnce: 1,
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
env: []string{
// cmd/go takes ~192 seconds on openbsd-386
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
})
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
addBuilder(BuildConfig{
Name: "openbsd-amd64-62",
HostType: "host-openbsd-amd64-62",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
dashboard: switch OpenBSD trybot from 6.2 to 6.4 OpenBSD 6.4 is the current OpenBSD release, while 6.2 has just stopped being supported. The new 6.4 builder has proven to be stable and reliable, so start using it for trybots as well. Looking at the non-trybot build times for the Go repository with OpenBSD 6.2 and 6.4 builders to get a sense of the build speed, it seems OpenBSD 6.4 is about 15% slower (1200~ seconds on 6.2 vs 1380~ seconds on 6.4). This might be due to security-oriented changes in OpenBSD 6.3 and 6.4 that were done to mitigate Spectre/Meltdown vulnerabilities. CL 145022 has helped recover some lost performance by re-enabling SMT. OpenBSD trybots seem to take around 6 minutes now, so there's some room to make them slower before they're the bottleneck. Try to update for now, and if it becomes a bottleneck, then we can prioritize investigating ways to improve OpenBSD 6.4 builder performance further if possible. References: • https://www.openbsd.org/64.html • https://www.openbsd.org/63.html • https://man.openbsd.org/sysctl.2#HW_SMT_2 Fixes golang/go#26833 Change-Id: I42c136fab800df46866a95017a8f70047b37f847 Reviewed-on: https://go-review.googlesource.com/c/144777 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 02:07:13 +03:00
tryBot: nil,
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
numTestHelpers: 0,
numTryTestHelpers: 5,
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 1,
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
})
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
addBuilder(BuildConfig{
Name: "openbsd-amd64-64",
HostType: "host-openbsd-amd64-64",
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
MinimumGoVersion: types.MajorMinor{1, 11},
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
dashboard: switch OpenBSD trybot from 6.2 to 6.4 OpenBSD 6.4 is the current OpenBSD release, while 6.2 has just stopped being supported. The new 6.4 builder has proven to be stable and reliable, so start using it for trybots as well. Looking at the non-trybot build times for the Go repository with OpenBSD 6.2 and 6.4 builders to get a sense of the build speed, it seems OpenBSD 6.4 is about 15% slower (1200~ seconds on 6.2 vs 1380~ seconds on 6.4). This might be due to security-oriented changes in OpenBSD 6.3 and 6.4 that were done to mitigate Spectre/Meltdown vulnerabilities. CL 145022 has helped recover some lost performance by re-enabling SMT. OpenBSD trybots seem to take around 6 minutes now, so there's some room to make them slower before they're the bottleneck. Try to update for now, and if it becomes a bottleneck, then we can prioritize investigating ways to improve OpenBSD 6.4 builder performance further if possible. References: • https://www.openbsd.org/64.html • https://www.openbsd.org/63.html • https://man.openbsd.org/sysctl.2#HW_SMT_2 Fixes golang/go#26833 Change-Id: I42c136fab800df46866a95017a8f70047b37f847 Reviewed-on: https://go-review.googlesource.com/c/144777 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 02:07:13 +03:00
tryBot: defaultTrySet(),
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
numTestHelpers: 0,
numTryTestHelpers: 5,
MaxAtOnce: 1,
})
addBuilder(BuildConfig{
Name: "openbsd-386-64",
HostType: "host-openbsd-386-64",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("sys"),
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
MaxAtOnce: 1,
})
dashboard: skip the "reboot" dist test on NetBSD due to disk space issues The NetBSD builders apparently have a tiny filesystem. And this test doesn't need to be run everywhere. A few builders is enough. Fixes golang/go#30839 Change-Id: I60df10173112ef9957c0e4cd6d414a4c5efaa940 Reviewed-on: https://go-review.googlesource.com/c/build/+/167637 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-14 18:55:17 +03:00
netBSDDistTestPolicy := func(distTest string, isTry bool) bool {
// Skip the test directory (slow, and adequately
// covered by other builders), and skip the "reboot"
// test, which takes more disk space on /tmp than the
// NetBSD image had as of 2019-03-13. (Issue 30839)
if strings.HasPrefix(distTest, "test:") || distTest == "reboot" {
return false
}
return true
}
cmd/coordinator/buildongce, dashboard: fix basepin creation, add NetBSD Updates golang/go#8643 Change-Id: I031ffb52537500c52df34f54552296c61dc2df10 Reviewed-on: https://go-review.googlesource.com/21493 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-04-04 21:03:49 +03:00
addBuilder(BuildConfig{
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
Name: "netbsd-amd64-8_0",
HostType: "host-netbsd-amd64-8_0",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: netBSDDistTestPolicy,
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
MaxAtOnce: 1,
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("sys"),
all: enable NetBSD 7.0 builders Fixes golang/go#8643 Change-Id: Id6a99d0bf4f032c5f5a2e5c282103cc98a515c7b Reviewed-on: https://go-review.googlesource.com/37469 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-26 21:11:21 +03:00
})
addBuilder(BuildConfig{
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
Name: "netbsd-386-8_0",
HostType: "host-netbsd-386-8_0",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: netBSDDistTestPolicy,
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
MaxAtOnce: 1,
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
// This builder currently hangs in the “../test” phase of all.bash.
// (https://golang.org/issue/25206)
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
buildsRepo: disabledBuilder,
cmd/coordinator/buildongce, dashboard: fix basepin creation, add NetBSD Updates golang/go#8643 Change-Id: I031ffb52537500c52df34f54552296c61dc2df10 Reviewed-on: https://go-review.googlesource.com/21493 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-04-04 21:03:49 +03:00
})
dashboard: add netbsd-arm-bsiegert This seems to be the last builder still using the old builder binary and not buildlet. It needs an entry in here to switch to buildlet and stage0. Change-Id: If2fdd2298ab88d56b6850061050caf27ca7c8983 Reviewed-on: https://go-review.googlesource.com/c/build/+/168061 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-19 16:28:04 +03:00
addBuilder(BuildConfig{
Name: "netbsd-arm-bsiegert",
HostType: "host-netbsd-arm-bsiegert",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: netBSDDistTestPolicy,
dashboard: add netbsd-arm-bsiegert This seems to be the last builder still using the old builder binary and not buildlet. It needs an entry in here to switch to buildlet and stage0. Change-Id: If2fdd2298ab88d56b6850061050caf27ca7c8983 Reviewed-on: https://go-review.googlesource.com/c/build/+/168061 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-19 16:28:04 +03:00
MaxAtOnce: 1,
tryBot: nil,
env: []string{
// The machine is slow.
"GO_TEST_TIMEOUT_SCALE=10",
},
})
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "plan9-386",
HostType: "host-plan9-386-gce",
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
numTestHelpers: 1,
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: func(distTestName string, isTry bool) bool {
env/plan9: revert SMP changes, stop running cmd/go tests Revert "env: enable SMP in Plan 9 image" This reverts commit 6478d3e7bdff4c8d810ba44f970ca4b0bfc92405. Revert "dashboard: bump plan9 GCE image" This reverts commit ce208996bf954a87d68789bb6a9000256f016c77. Updates golang/go#29822 Change-Id: I47e75d135e092ee9d5a80c66e91c1048c67cd5f6 Reviewed-on: https://go-review.googlesource.com/c/158542 Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: David du Colombier <0intro@gmail.com>
2019-01-19 01:19:58 +03:00
switch distTestName {
case "api",
"go_test:cmd/go": // takes over 20 minutes without working SMP
return false
}
return true
},
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
})
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "windows-amd64-2008",
HostType: "host-windows-amd64-2008",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: onlyGo,
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
})
cmd/buildlet, dashboard: support for v2 windows images, 386 builder - images now include 32bit gcc in c:\godeps\gcc32 - 64bit gcc has moved to c:\godeps\gcc64 - update host definitions for new image versions - add windows-386-2008 host Updates golang/go#17513 Change-Id: Ib3f5d4b2b1702c05089cf633529ebf38db270a43 Reviewed-on: https://go-review.googlesource.com/42014 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-27 23:12:08 +03:00
addBuilder(BuildConfig{
Name: "windows-386-2008",
HostType: "host-windows-amd64-2008",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: fasterTrybots,
cmd/buildlet, dashboard: support for v2 windows images, 386 builder - images now include 32bit gcc in c:\godeps\gcc32 - 64bit gcc has moved to c:\godeps\gcc64 - update host definitions for new image versions - add windows-386-2008 host Updates golang/go#17513 Change-Id: Ib3f5d4b2b1702c05089cf633529ebf38db270a43 Reviewed-on: https://go-review.googlesource.com/42014 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-27 23:12:08 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
numTryTestHelpers: 4,
cmd/buildlet, dashboard: support for v2 windows images, 386 builder - images now include 32bit gcc in c:\godeps\gcc32 - 64bit gcc has moved to c:\godeps\gcc64 - update host definitions for new image versions - add windows-386-2008 host Updates golang/go#17513 Change-Id: Ib3f5d4b2b1702c05089cf633529ebf38db270a43 Reviewed-on: https://go-review.googlesource.com/42014 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-27 23:12:08 +03:00
})
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "windows-amd64-2012",
HostType: "host-windows-amd64-2012",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: onlyGo,
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
MaxAtOnce: 2,
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
})
build: use buildlet on Windows Also adds "gomote rmall" command. Fixes golang/go#8640 Fixes golang/go#9799 Change-Id: I41b62b16df5e855e1367a9cc1c370bb1f928c242 Reviewed-on: https://go-review.googlesource.com/4132 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2015-02-07 04:32:15 +03:00
addBuilder(BuildConfig{
dashboard: update Windows builders, remove old manual Windows builder And clean up some other builders. Change-Id: I1f71836da7008e58d3e76e2cc3170e96cd57ddf6 Reviewed-on: https://go-review.googlesource.com/51432 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-27 07:52:12 +03:00
Name: "windows-amd64-2016",
HostType: "host-windows-amd64-2016",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: fasterTrybots,
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
dashboard: update Windows builders, remove old manual Windows builder And clean up some other builders. Change-Id: I1f71836da7008e58d3e76e2cc3170e96cd57ddf6 Reviewed-on: https://go-review.googlesource.com/51432 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-27 07:52:12 +03:00
numTryTestHelpers: 5,
build: use buildlet on Windows Also adds "gomote rmall" command. Fixes golang/go#8640 Fixes golang/go#9799 Change-Id: I41b62b16df5e855e1367a9cc1c370bb1f928c242 Reviewed-on: https://go-review.googlesource.com/4132 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2015-02-07 04:32:15 +03:00
})
addBuilder(BuildConfig{
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
Name: "windows-amd64-race",
HostType: "host-windows-amd64-2008",
Notes: "Only runs -race tests (./race.bat)",
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2"},
build: use buildlet on Windows Also adds "gomote rmall" command. Fixes golang/go#8640 Fixes golang/go#9799 Change-Id: I41b62b16df5e855e1367a9cc1c370bb1f928c242 Reviewed-on: https://go-review.googlesource.com/4132 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2015-02-07 04:32:15 +03:00
})
dashboard: add windows-arm builder Updates golang/go#26148 Change-Id: I23a2c7bd1c2f8f7c960aaedfe0dcdee4e11825ab Reviewed-on: https://go-review.googlesource.com/125643 Reviewed-by: Andrew Bonventre <andybons@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-24 22:59:34 +03:00
addBuilder(BuildConfig{
dashboard: skip writing snapshot for windows/arm builder Writing the snapshot to GCS is timing out for the Windows/ARM builder. It tars about 500MB of data, resulting in about 175MB of data, which it then sends to GCS. It takes about 4:30 to tar the workdir/go directory with the built-in tar command, leaving only 30s to transfer 175MB across the web, so it's not surprising that it times out. Updates golang/go#26148 Change-Id: I7e70aeaf1ac6334c17fdca64ae9ab946c3d87d9a Reviewed-on: https://go-review.googlesource.com/c/153518 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-12-11 03:56:05 +03:00
Name: "windows-arm",
HostType: "host-windows-arm-iotcore",
SkipSnapshot: true,
dashboard: add windows-arm builder Updates golang/go#26148 Change-Id: I23a2c7bd1c2f8f7c960aaedfe0dcdee4e11825ab Reviewed-on: https://go-review.googlesource.com/125643 Reviewed-by: Andrew Bonventre <andybons@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-24 22:59:34 +03:00
env: []string{
"GOARM=7",
"GO_TEST_TIMEOUT_SCALE=2",
},
})
dashboard: add OS X 10.8 and 10.11 reverse builders Updates golang/go#9495 Updates golang/go#12185 Change-Id: I84548011823334f0876636f8e9685e07238cafc4 Reviewed-on: https://go-review.googlesource.com/28171 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 22:55:08 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "darwin-amd64-10_8",
HostType: "host-darwin-10_8",
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
shouldRunDistTest: macTestPolicy,
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
buildsRepo: disabledBuilder,
dashboard: add OS X 10.8 and 10.11 reverse builders Updates golang/go#9495 Updates golang/go#12185 Change-Id: I84548011823334f0876636f8e9685e07238cafc4 Reviewed-on: https://go-review.googlesource.com/28171 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 22:55:08 +03:00
})
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "darwin-amd64-10_10",
HostType: "host-darwin-10_10",
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
shouldRunDistTest: macTestPolicy,
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
// https://tip.golang.org/doc/go1.12 says:
// "Go 1.12 is the last release that will run on macOS 10.10 Yosemite."
major, minor, ok := version.ParseReleaseBranch(branch)
return repo == "go" && ok && major == 1 && minor <= 12
},
dashboard: add OS X 10.8 and 10.11 reverse builders Updates golang/go#9495 Updates golang/go#12185 Change-Id: I84548011823334f0876636f8e9685e07238cafc4 Reviewed-on: https://go-review.googlesource.com/28171 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 22:55:08 +03:00
})
cmd/coordinator, dashboard: switch to new VMWare-based OS X builders Ignore the old darwin-{amd64,386}-10_10 builders. Don't give them an error, but pretend they don't exist. Also: switch trybots from OS X 10.10 to OS X 10.11, and re-enable sharding. Let's hope for the best. See golang/go#12979. This also enables subrepo tests for all OS X versions. darwin-386-* is currently offline, pending some golang/go#17009 Updates golang/go#9495 (OS X virtualization) Change-Id: I4d53a79087404b5e8051d1aff0c668a92625f442 Reviewed-on: https://go-review.googlesource.com/28583 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-07 08:41:02 +03:00
addBuilder(BuildConfig{
dashboard: fix typo from CL 94601 Sigh. Change-Id: Ife6006f9b0c65fa6ee7be50b6c9f55319e0fa28d Reviewed-on: https://go-review.googlesource.com/94602 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-02-16 02:43:31 +03:00
Name: "darwin-amd64-10_11",
HostType: "host-darwin-10_11",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil, // disabled until Macs fixed; https://golang.org/issue/23859
dashboard: don't test subrepos on both macOS 10.11 and 10.12 Just the latter. Reduce resource usage on limited Macs. Change-Id: Ia282627235c39d74adc9e059d8f39ed8e770473a Reviewed-on: https://go-review.googlesource.com/c/build/+/167999 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-18 19:48:22 +03:00
buildsRepo: onlyGo,
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
shouldRunDistTest: macTestPolicy,
dashboard: fix typo from CL 94601 Sigh. Change-Id: Ife6006f9b0c65fa6ee7be50b6c9f55319e0fa28d Reviewed-on: https://go-review.googlesource.com/94602 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-02-16 02:43:31 +03:00
numTryTestHelpers: 3,
cmd/coordinator, dashboard: switch to new VMWare-based OS X builders Ignore the old darwin-{amd64,386}-10_10 builders. Don't give them an error, but pretend they don't exist. Also: switch trybots from OS X 10.10 to OS X 10.11, and re-enable sharding. Let's hope for the best. See golang/go#12979. This also enables subrepo tests for all OS X versions. darwin-386-* is currently offline, pending some golang/go#17009 Updates golang/go#9495 (OS X virtualization) Change-Id: I4d53a79087404b5e8051d1aff0c668a92625f442 Reviewed-on: https://go-review.googlesource.com/28583 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-07 08:41:02 +03:00
})
dashboard: restore darwin-386 buidler Fixes golang/go#17009 Change-Id: Ia9008bcb7b47b7671c158c1a00afc4e765805e4e Reviewed-on: https://go-review.googlesource.com/41617 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-25 02:03:42 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "darwin-386-10_11",
HostType: "host-darwin-10_11",
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
shouldRunDistTest: macTestPolicy,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: onlyGo,
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
MaxAtOnce: 1,
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
})
addBuilder(BuildConfig{
Name: "darwin-amd64-10_12",
HostType: "host-darwin-10_12",
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
shouldRunDistTest: macTestPolicy,
dashboard: restore darwin-386 buidler Fixes golang/go#17009 Change-Id: Ia9008bcb7b47b7671c158c1a00afc4e765805e4e Reviewed-on: https://go-review.googlesource.com/41617 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-25 02:03:42 +03:00
})
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
addBuilder(BuildConfig{
Name: "darwin-amd64-10_14",
HostType: "host-darwin-10_14",
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
shouldRunDistTest: macTestPolicy,
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
})
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "darwin-amd64-race",
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
HostType: "host-darwin-10_12",
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
shouldRunDistTest: macTestPolicy,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: onlyGo,
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
})
dashboard: add zenly as builder for ios (Iphone 5C, 7+) Change-Id: Ie1dd5c53ed0b2d207297ee0cd3ba46a111167e30 Reviewed-on: https://go-review.googlesource.com/125815 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-25 16:14:32 +03:00
addBuilder(BuildConfig{
Name: "darwin-arm-mg912baios",
HostType: "host-darwin-amd64-zenly-ios",
Notes: "iPhone 5C (model MG912B/A), via a MacBook Pro; owned by zenly",
env: []string{
"GOARCH=arm",
"GOIOS_DEVICE_ID=8e5c23a5d0843d1ffe164ea0b2f2500599c3ebff",
},
})
addBuilder(BuildConfig{
Name: "darwin-arm64-mn4m2zdaios",
HostType: "host-darwin-amd64-zenly-ios",
Notes: "iPhone 7+ (model MN4M2ZD/A), via a MacBook Pro; owned by zenly",
env: []string{
"GOARCH=arm64",
"GOIOS_DEVICE_ID=5ec20fafe317e1c8ff51efc6d508cf19808474a2",
},
})
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
addBuilder(BuildConfig{
dashboard: add x/mobile builder that runs on the host with device attached When running GOOS=android go test <pkg>, the go tool will build the test(s) for Android and run it through the exec wrapper. The exec wrappers takes care of pushing the binary to the device and running it. That's how the normal builder with an Android device attached works: setting GOOS=android. Some x/mobile tests uses the Android SDK and then use an Android specific way to run integration tests with Java. Those tests don't run on GOOS=android. To solve that dilemma, define a special builder without GOOS=android set and use it (only) for x/mobile. Then, disable the GOOS=android builder for x/mobile to avoid wasting device time better spent on the main repository. Similar to CL 164740. While here, correct two GOHOSTOS values and rename the existing android builders to match the GOOS-GOARCH-suffix form. Updates golang/go#30482 Change-Id: I3712584c62c3765d5605cccb642aa6f119826ae8 Reviewed-on: https://go-review.googlesource.com/c/build/+/164660 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-01 23:09:34 +03:00
Name: "darwin-amd64-wikofever",
dashboard: split mobile builders into iOS and Android Even though both mobile builders will run off the same host Mac Mini, Android and iOS builds can run simultaneously. Split the eliasnaur host in two to allow that. Change-Id: If3db4dadb56b8a4b212dcbb3015e3e6c1b69608d Reviewed-on: https://go-review.googlesource.com/39130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-31 11:08:01 +03:00
HostType: "host-darwin-amd64-eliasnaur-android",
dashboard: add x/mobile builder that runs on the host with device attached When running GOOS=android go test <pkg>, the go tool will build the test(s) for Android and run it through the exec wrapper. The exec wrappers takes care of pushing the binary to the device and running it. That's how the normal builder with an Android device attached works: setting GOOS=android. Some x/mobile tests uses the Android SDK and then use an Android specific way to run integration tests with Java. Those tests don't run on GOOS=android. To solve that dilemma, define a special builder without GOOS=android set and use it (only) for x/mobile. Then, disable the GOOS=android builder for x/mobile to avoid wasting device time better spent on the main repository. Similar to CL 164740. While here, correct two GOHOSTOS values and rename the existing android builders to match the GOOS-GOARCH-suffix form. Updates golang/go#30482 Change-Id: I3712584c62c3765d5605cccb642aa6f119826ae8 Reviewed-on: https://go-review.googlesource.com/c/build/+/164660 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-01 23:09:34 +03:00
Notes: "Same as android-arm*-wikofever but without GOOS set, for running x/mobile tests.",
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
return repo == "mobile" && branch == "master" && goBranch == "master"
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
dashboard: add x/mobile builder that runs on the host with device attached When running GOOS=android go test <pkg>, the go tool will build the test(s) for Android and run it through the exec wrapper. The exec wrappers takes care of pushing the binary to the device and running it. That's how the normal builder with an Android device attached works: setting GOOS=android. Some x/mobile tests uses the Android SDK and then use an Android specific way to run integration tests with Java. Those tests don't run on GOOS=android. To solve that dilemma, define a special builder without GOOS=android set and use it (only) for x/mobile. Then, disable the GOOS=android builder for x/mobile to avoid wasting device time better spent on the main repository. Similar to CL 164740. While here, correct two GOHOSTOS values and rename the existing android builders to match the GOOS-GOARCH-suffix form. Updates golang/go#30482 Change-Id: I3712584c62c3765d5605cccb642aa6f119826ae8 Reviewed-on: https://go-review.googlesource.com/c/build/+/164660 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-01 23:09:34 +03:00
env: []string{
"CGO_ENABLED=1",
},
})
addBuilder(BuildConfig{
Name: "android-arm-wikofever",
HostType: "host-darwin-amd64-eliasnaur-android",
Notes: "Android Wiko Fever phone running Android 6.0, via a Mac Mini",
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
return repo == "go" && branch == "master" && goBranch == "master"
},
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
env: []string{
"GOARCH=arm",
dashboard: remove eliasnaur Android emulators Now that we hace Android emulator builders on GCE, drop the old ones running on hardware. Also update the configuration of the remaining builders now that androidtest.bash is no longer necessary. Change-Id: I09cab08b28952f967205d6c35f36eb6e05a17950 Reviewed-on: https://go-review.googlesource.com/c/164397 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 10:19:36 +03:00
"GOOS=android",
"GOHOSTARCH=amd64",
dashboard: add x/mobile builder that runs on the host with device attached When running GOOS=android go test <pkg>, the go tool will build the test(s) for Android and run it through the exec wrapper. The exec wrappers takes care of pushing the binary to the device and running it. That's how the normal builder with an Android device attached works: setting GOOS=android. Some x/mobile tests uses the Android SDK and then use an Android specific way to run integration tests with Java. Those tests don't run on GOOS=android. To solve that dilemma, define a special builder without GOOS=android set and use it (only) for x/mobile. Then, disable the GOOS=android builder for x/mobile to avoid wasting device time better spent on the main repository. Similar to CL 164740. While here, correct two GOHOSTOS values and rename the existing android builders to match the GOOS-GOARCH-suffix form. Updates golang/go#30482 Change-Id: I3712584c62c3765d5605cccb642aa6f119826ae8 Reviewed-on: https://go-review.googlesource.com/c/build/+/164660 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-01 23:09:34 +03:00
"GOHOSTOS=darwin",
dashboard: remove eliasnaur Android emulators Now that we hace Android emulator builders on GCE, drop the old ones running on hardware. Also update the configuration of the remaining builders now that androidtest.bash is no longer necessary. Change-Id: I09cab08b28952f967205d6c35f36eb6e05a17950 Reviewed-on: https://go-review.googlesource.com/c/164397 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 10:19:36 +03:00
"CGO_ENABLED=1",
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
},
})
addBuilder(BuildConfig{
dashboard: add x/mobile builder that runs on the host with device attached When running GOOS=android go test <pkg>, the go tool will build the test(s) for Android and run it through the exec wrapper. The exec wrappers takes care of pushing the binary to the device and running it. That's how the normal builder with an Android device attached works: setting GOOS=android. Some x/mobile tests uses the Android SDK and then use an Android specific way to run integration tests with Java. Those tests don't run on GOOS=android. To solve that dilemma, define a special builder without GOOS=android set and use it (only) for x/mobile. Then, disable the GOOS=android builder for x/mobile to avoid wasting device time better spent on the main repository. Similar to CL 164740. While here, correct two GOHOSTOS values and rename the existing android builders to match the GOOS-GOARCH-suffix form. Updates golang/go#30482 Change-Id: I3712584c62c3765d5605cccb642aa6f119826ae8 Reviewed-on: https://go-review.googlesource.com/c/build/+/164660 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-01 23:09:34 +03:00
Name: "android-arm64-wikofever",
dashboard: split mobile builders into iOS and Android Even though both mobile builders will run off the same host Mac Mini, Android and iOS builds can run simultaneously. Split the eliasnaur host in two to allow that. Change-Id: If3db4dadb56b8a4b212dcbb3015e3e6c1b69608d Reviewed-on: https://go-review.googlesource.com/39130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-31 11:08:01 +03:00
HostType: "host-darwin-amd64-eliasnaur-android",
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
Notes: "Android Wiko Fever phone running Android 6.0, via a Mac Mini",
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
return repo == "go" && branch == "master" && goBranch == "master"
},
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
env: []string{
"GOARCH=arm64",
dashboard: remove eliasnaur Android emulators Now that we hace Android emulator builders on GCE, drop the old ones running on hardware. Also update the configuration of the remaining builders now that androidtest.bash is no longer necessary. Change-Id: I09cab08b28952f967205d6c35f36eb6e05a17950 Reviewed-on: https://go-review.googlesource.com/c/164397 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 10:19:36 +03:00
"GOOS=android",
"GOHOSTARCH=amd64",
dashboard: add x/mobile builder that runs on the host with device attached When running GOOS=android go test <pkg>, the go tool will build the test(s) for Android and run it through the exec wrapper. The exec wrappers takes care of pushing the binary to the device and running it. That's how the normal builder with an Android device attached works: setting GOOS=android. Some x/mobile tests uses the Android SDK and then use an Android specific way to run integration tests with Java. Those tests don't run on GOOS=android. To solve that dilemma, define a special builder without GOOS=android set and use it (only) for x/mobile. Then, disable the GOOS=android builder for x/mobile to avoid wasting device time better spent on the main repository. Similar to CL 164740. While here, correct two GOHOSTOS values and rename the existing android builders to match the GOOS-GOARCH-suffix form. Updates golang/go#30482 Change-Id: I3712584c62c3765d5605cccb642aa6f119826ae8 Reviewed-on: https://go-review.googlesource.com/c/build/+/164660 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-01 23:09:34 +03:00
"GOHOSTOS=darwin",
dashboard: remove eliasnaur Android emulators Now that we hace Android emulator builders on GCE, drop the old ones running on hardware. Also update the configuration of the remaining builders now that androidtest.bash is no longer necessary. Change-Id: I09cab08b28952f967205d6c35f36eb6e05a17950 Reviewed-on: https://go-review.googlesource.com/c/164397 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 10:19:36 +03:00
"CGO_ENABLED=1",
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
},
})
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
addBuilder(BuildConfig{
Name: "android-386-emu",
HostType: "host-android-amd64-emu", // same amd64 host is used for 386 builder
Notes: "Android emulator on GCE",
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
switch repo {
case "blog", "talks", "review", "tour":
return false
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
return atLeastGo1(branch, 13) && atLeastGo1(goBranch, 13)
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
env: []string{
"GOARCH=386",
"GOOS=android",
"GOHOSTARCH=amd64",
"GOHOSTOS=linux",
"CGO_ENABLED=1",
},
})
addBuilder(BuildConfig{
Name: "android-amd64-emu",
HostType: "host-android-amd64-emu",
Notes: "Android emulator on GCE",
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
tryBot: func(repo, branch, goBranch string) bool {
return repo == "mobile" && atLeastGo1(branch, 13) && atLeastGo1(goBranch, 13)
cmd/coordinator: test x/mobile on android emu builders; and exp on some Updates golang/go#23824 Change-Id: Iee70bcbfef6a271fdd26694d848a703a200dc0c0 Reviewed-on: https://go-review.googlesource.com/c/164478 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 20:32:45 +03:00
},
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
switch repo {
case "blog", "talks", "review", "tour":
return false
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
return atLeastGo1(branch, 13) && atLeastGo1(goBranch, 13)
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
env: []string{
"GOARCH=amd64",
"GOOS=android",
"GOHOSTARCH=amd64",
"GOHOSTOS=linux",
"CGO_ENABLED=1",
},
})
x/build: add Solaris amd64 release builder to dashboard Fixes golang/go#20949 Change-Id: I4f85ee76783a1240a2ebd4a2a1f167ab849fcebe Reviewed-on: https://go-review.googlesource.com/47811 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-07 23:41:20 +03:00
addBuilder(BuildConfig{
Name: "solaris-amd64-oraclerel",
HostType: "host-solaris-oracle-amd64-oraclerel",
Notes: "Oracle Solaris release version",
})
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "solaris-amd64-smartosbuildlet",
HostType: "host-solaris-amd64",
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
})
all: add ppc64 (big endian) builders Change-Id: I0fddda33179555a50b6f03dcff13338544b742c3 Reviewed-on: https://go-review.googlesource.com/30599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-06 23:30:58 +03:00
addBuilder(BuildConfig{
Name: "linux-ppc64-buildlet",
HostType: "host-linux-ppc64-osu",
FlakyNet: true,
})
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "linux-ppc64le-buildlet",
HostType: "host-linux-ppc64le-osu",
FlakyNet: true,
dashboard: add arm64 buildlet, update ppc64 buildlet definition Change-Id: I25c6de0f97c3338f9d32f4125b2c7c68c3f3c13c Reviewed-on: https://go-review.googlesource.com/21656 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2016-04-07 22:27:15 +03:00
})
dashboard: add linux-ppc64le POWER9 builder This change adds an extra linux-ppc64le builder running on POWER9. Fixes golang/go#30138 Change-Id: I23bdfc116bb89c6d96be330c2c87827c47705b6b Reviewed-on: https://go-review.googlesource.com/c/163757 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:05:12 +03:00
addBuilder(BuildConfig{
Name: "linux-ppc64le-power9osu",
HostType: "host-linux-ppc64le-power9-osu",
FlakyNet: true,
})
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
addBuilder(BuildConfig{
Name: "linux-arm64-packet",
HostType: "host-linux-arm64-packet",
all: remove config for arm64 linaro builders They've been decomissioned for some months now. Our arm64 builders are on packet.net these days. Change-Id: Iaede189157e0298812b2971cdd63599f6096d73d Reviewed-on: https://go-review.googlesource.com/c/build/+/165781 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 08:16:49 +03:00
FlakyNet: true, // maybe not flaky, but here conservatively
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
})
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
addBuilder(BuildConfig{
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
Name: "linux-mips",
HostType: "host-linux-mips",
SkipSnapshot: true,
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
})
addBuilder(BuildConfig{
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
Name: "linux-mipsle",
HostType: "host-linux-mipsle",
SkipSnapshot: true,
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
})
addBuilder(BuildConfig{
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
Name: "linux-mips64",
HostType: "host-linux-mips64",
SkipSnapshot: true,
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
})
addBuilder(BuildConfig{
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
Name: "linux-mips64le",
HostType: "host-linux-mips64le",
SkipSnapshot: true,
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
})
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
addBuilder(BuildConfig{
Name: "linux-s390x-ibm",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
HostType: "host-linux-s390x",
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
numTestHelpers: 0,
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
})
dashboard: add s390x cross-compile-only builder and release config Also modifies cmd/release to wire it up. Updates golang/go#16443 Change-Id: Ie80a224957f6f662d0aec0fd083e2a57e30436c4 Reviewed-on: https://go-review.googlesource.com/28172 Reviewed-by: Jessica Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 23:32:33 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "linux-s390x-crosscompile",
HostType: "host-s390x-cross-kube",
Notes: "s390x cross-compile builder for releases; doesn't run tests",
CompileOnly: true,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
tryOnly: true, // but not in trybot set for now
dashboard: add s390x cross-compile-only builder and release config Also modifies cmd/release to wire it up. Updates golang/go#16443 Change-Id: Ie80a224957f6f662d0aec0fd083e2a57e30436c4 Reviewed-on: https://go-review.googlesource.com/28172 Reviewed-by: Jessica Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 23:32:33 +03:00
env: []string{
"CGO_ENABLED=1",
"GOARCH=s390x",
"GOHOSTARCH=amd64",
"CC_FOR_TARGET=s390x-linux-gnu-gcc",
},
})
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-localdev",
HostType: "host-linux-amd64-localdev",
Notes: "for localhost development only",
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
tryOnly: true,
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
})
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "dragonfly-amd64",
HostType: "host-dragonfly-amd64-tdfbsd",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
SkipSnapshot: true,
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
})
addBuilder(BuildConfig{
builders: skip reboot test on freebsd-arm Change-Id: I77822252a73d61c240c503bc424f03e5cfc6a95a Reviewed-on: https://go-review.googlesource.com/c/build/+/168064 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-20 22:00:27 +03:00
Name: "freebsd-arm-paulzhol",
HostType: "host-freebsd-arm-paulzhol",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
builders: skip reboot test on freebsd-arm Change-Id: I77822252a73d61c240c503bc424f03e5cfc6a95a Reviewed-on: https://go-review.googlesource.com/c/build/+/168064 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-20 22:00:27 +03:00
SkipSnapshot: true,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
// This was a fragile little machine with limited memory.
// Only run a few of the core subrepos for now while
// we figure out what's killing it.
switch repo {
case "go", "sys", "net":
return true
}
return false
},
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
env: []string{
"GOARM=7",
"CGO_ENABLED=1",
},
})
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
addBuilder(BuildConfig{
builders: skip the reboot test in more places, especially on slower builders It's not important to run the new "reboot" bootstrapping test on all builders; a few is fine. Skip it on plan9-arm, etc, and on trybots that aren't super widely sharded. Change-Id: I35d5408eb5e034f42df5976f2b9b7f7d478d580a Reviewed-on: https://go-review.googlesource.com/c/build/+/167998 Reviewed-by: David du Colombier <0intro@gmail.com>
2019-03-18 01:28:15 +03:00
Name: "plan9-arm",
HostType: "host-plan9-arm-0intro",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
builders: skip the reboot test in more places, especially on slower builders It's not important to run the new "reboot" bootstrapping test on all builders; a few is fine. Skip it on plan9-arm, etc, and on trybots that aren't super widely sharded. Change-Id: I35d5408eb5e034f42df5976f2b9b7f7d478d580a Reviewed-on: https://go-review.googlesource.com/c/build/+/167998 Reviewed-by: David du Colombier <0intro@gmail.com>
2019-03-18 01:28:15 +03:00
SkipSnapshot: true,
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
})
addBuilder(BuildConfig{
Name: "nacl-arm",
HostType: "host-nacl-arm-davecheney",
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
buildsRepo: onlyGo,
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
SkipSnapshot: true,
})
addBuilder(BuildConfig{
builders: skip the reboot test in more places, especially on slower builders It's not important to run the new "reboot" bootstrapping test on all builders; a few is fine. Skip it on plan9-arm, etc, and on trybots that aren't super widely sharded. Change-Id: I35d5408eb5e034f42df5976f2b9b7f7d478d580a Reviewed-on: https://go-review.googlesource.com/c/build/+/167998 Reviewed-by: David du Colombier <0intro@gmail.com>
2019-03-18 01:28:15 +03:00
Name: "plan9-amd64-9front",
HostType: "host-plan9-amd64-0intro",
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
shouldRunDistTest: noTestDir,
builders: skip the reboot test in more places, especially on slower builders It's not important to run the new "reboot" bootstrapping test on all builders; a few is fine. Skip it on plan9-arm, etc, and on trybots that aren't super widely sharded. Change-Id: I35d5408eb5e034f42df5976f2b9b7f7d478d580a Reviewed-on: https://go-review.googlesource.com/c/build/+/167998 Reviewed-by: David du Colombier <0intro@gmail.com>
2019-03-18 01:28:15 +03:00
SkipSnapshot: true,
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
})
dashboard: add aix-ppc64 builder This is a starting point for adding an aix-ppc64 reverse builder. There can be further adjustments made to it as needed. Updates golang/go#27160. Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508 Reviewed-on: https://go-review.googlesource.com/131335 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-24 21:41:05 +03:00
addBuilder(BuildConfig{
dashboard: require at least Go 1.12 for aix builder Fixes golang/go#30799 Change-Id: I0b3b53b715e2dc00f10548f74443224190003818 Reviewed-on: https://go-review.googlesource.com/c/build/+/167357 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 11:29:15 +03:00
Name: "aix-ppc64",
HostType: "host-aix-ppc64-osuosl",
MinimumGoVersion: types.MajorMinor{1, 12},
dashboard: add /opt/freeware/bin to PATH on aix builder Updates golang/go#29065 Fixes golang/go#29078 Change-Id: Ifa9355c9dc988a460b6198913431647ec2c5e6ac Reviewed-on: https://go-review.googlesource.com/c/152457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-12-04 17:46:08 +03:00
env: []string{
"PATH=/opt/freeware/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin",
},
dashboard: add aix-ppc64 builder This is a starting point for adding an aix-ppc64 reverse builder. There can be further adjustments made to it as needed. Updates golang/go#27160. Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508 Reviewed-on: https://go-review.googlesource.com/131335 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-24 21:41:05 +03:00
})
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
}
dashboard: add s390x cross-compile-only builder and release config Also modifies cmd/release to wire it up. Updates golang/go#16443 Change-Id: Ie80a224957f6f662d0aec0fd083e2a57e30436c4 Reviewed-on: https://go-review.googlesource.com/28172 Reviewed-by: Jessica Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 23:32:33 +03:00
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
// addBuilder adds c to the Builders map after doing some sanity
// checks.
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
func addBuilder(c BuildConfig) {
if c.Name == "" {
panic("empty name")
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
if c.HostType == "" {
panic(fmt.Sprintf("missing HostType for builder %q", c.Name))
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
if _, dup := Builders[c.Name]; dup {
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
panic("dup name " + c.Name)
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
if _, ok := Hosts[c.HostType]; !ok {
panic(fmt.Sprintf("undefined HostType %q for builder %q", c.HostType, c.Name))
cmd/coordinator: start of a Kubernetes buildlet pool Updates golang/go#12546 Change-Id: Ia70baeb1df441509f70c77e7b94e7eb9fddccbf3 Reviewed-on: https://go-review.googlesource.com/14392 Reviewed-by: Evan Brown <evanbrown@google.com>
2015-09-09 01:18:47 +03:00
}
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
if c.SkipSnapshot && (c.numTestHelpers > 0 || c.numTryTestHelpers > 0) {
panic(fmt.Sprintf("config %q's SkipSnapshot is not compatible with sharded test helpers", c.Name))
}
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
types := 0
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
for _, fn := range []func() bool{c.IsReverse, c.IsContainer, c.IsVM} {
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
if fn() {
types++
}
}
if types != 1 {
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
panic(fmt.Sprintf("build config %q host type inconsistent (must be Reverse, Image, or VM)", c.Name))
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
}
all: change dashboard.BuildConfig to be a pointer type, like HostConfig BuildConfig grew way too large and has too many slices & such to be a value type. Make it a pointer type, which matches HostConfig. Change-Id: Ie625bece9d6d8c1ec6cff26e77416bd9b1f256d8 Reviewed-on: https://go-review.googlesource.com/c/145077 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 20:52:25 +03:00
Builders[c.Name] = &c
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
}
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
// fasterTrybots is a shouldRunDistTest policy function.
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
// It skips (returns false) the test/ directory tests for trybots.
func fasterTrybots(distTest string, isTry bool) bool {
builders: skip the reboot test in more places, especially on slower builders It's not important to run the new "reboot" bootstrapping test on all builders; a few is fine. Skip it on plan9-arm, etc, and on trybots that aren't super widely sharded. Change-Id: I35d5408eb5e034f42df5976f2b9b7f7d478d580a Reviewed-on: https://go-review.googlesource.com/c/build/+/167998 Reviewed-by: David du Colombier <0intro@gmail.com>
2019-03-18 01:28:15 +03:00
if isTry {
if strings.HasPrefix(distTest, "test:") ||
distTest == "reboot" {
return false // skip test
}
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
}
return true
}
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
// noTestDir is a shouldRunDistTest policy function.
builders: skip the reboot test in more places, especially on slower builders It's not important to run the new "reboot" bootstrapping test on all builders; a few is fine. Skip it on plan9-arm, etc, and on trybots that aren't super widely sharded. Change-Id: I35d5408eb5e034f42df5976f2b9b7f7d478d580a Reviewed-on: https://go-review.googlesource.com/c/build/+/167998 Reviewed-by: David du Colombier <0intro@gmail.com>
2019-03-18 01:28:15 +03:00
// It skips (returns false) the test/ directory tests for all builds,
// as well as the "reboot" test that tests that recompiling Go with
// the just-built Go works.
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
func noTestDir(distTest string, isTry bool) bool {
builders: skip the reboot test in more places, especially on slower builders It's not important to run the new "reboot" bootstrapping test on all builders; a few is fine. Skip it on plan9-arm, etc, and on trybots that aren't super widely sharded. Change-Id: I35d5408eb5e034f42df5976f2b9b7f7d478d580a Reviewed-on: https://go-review.googlesource.com/c/build/+/167998 Reviewed-by: David du Colombier <0intro@gmail.com>
2019-03-18 01:28:15 +03:00
if strings.HasPrefix(distTest, "test:") || distTest == "reboot" {
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
return false // skip test
}
return true
}
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
// TryBuildersForProject returns the builders that should run as part of
// a TryBot set for the given project.
// The project argument is of the form "go", "net", "sys", etc.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// The branch is the branch of that project ("master", "release-branch.go1.12", etc)
// The goBranch is the branch of Go to use. If proj == "go", then branch == goBranch.
func TryBuildersForProject(proj, branch, goBranch string) []*BuildConfig {
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
var confs []*BuildConfig
for _, conf := range Builders {
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
if conf.BuildsRepoTryBot(proj, branch, goBranch) {
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
confs = append(confs, conf)
}
}
sort.Slice(confs, func(i, j int) bool {
return confs[i].Name < confs[j].Name
})
return confs
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// atLeastGo1 reports whether branch is either "master" or "release-branch.go1.N" where N >= min.
func atLeastGo1(branch string, min int) bool {
if branch == "master" {
return true
}
major, minor, ok := version.ParseReleaseBranch(branch)
return ok && major == 1 && minor >= min
}
// onlyGo is a common buildsRepo policy value that only builds the main "go" repo.
func onlyGo(repo, branch, goBranch string) bool { return repo == "go" }
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
// disabledBuilder is a buildsRepo policy function that always return false.
func disabledBuilder(repo, branch, goBranch string) bool { return false }
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
// macTestPolicy is the test policy for Macs.
//
// We have limited Mac resources. It's not worth wasting time testing
// portable things on them. That is, if there's a slow test that will
// still fail slowly on another builder where we have more resources
// (like linux-amd64), then there's no point testing it redundantly on
// the Macs.
func macTestPolicy(distTest string, isTry bool) bool {
if strings.HasPrefix(distTest, "test:") {
return false
}
switch distTest {
case "reboot", "api", "doc_progs",
"wiki", "bench_go1", "codewalk":
return false
}
if isTry {
switch distTest {
case "runtime:cpu124", "race", "moved_goroot":
return false
}
// TODO: more. Look at bigquery results once we have more data.
}
return true
}