go
/
build
зеркало из https://github.com/golang/build.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
build/dashboard/builders.go

2001 строка
66 KiB
Go
Исходник Обычный вид История

dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
// Copyright 2015 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package dashboard contains shared configuration and logic used by various
// pieces of the Go continuous build system.
package dashboard
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
import (
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"fmt"
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
"os"
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
"sort"
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
"strconv"
"strings"
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
"time"
all: consolidate configuration for coordinator and gomote buildenv.Environment type defines configuration options: - Coordinator uses the GCE project name to lookup config. A custom config name can be provided at runtime to override. - The conventional prod and stage project names ('symbolic-datum-552' and 'go-dashboard-dev') map to prod and staging configuration structs. - Production and staging status is explicitly defined in configuration. - GCS bucket names for buildlet, logs, and snapshots are configurable. Change-Id: I7e6d7874eb0bdfe35dbdd5fcf6212ab50d576b88 Reviewed-on: https://go-review.googlesource.com/19502 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-15 02:59:59 +03:00
"golang.org/x/build/buildenv"
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
"golang.org/x/build/types"
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
)
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
// Builders are the different build configurations.
// The keys are like "darwin-amd64" or "linux-386-387".
// This map should not be modified by other packages.
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
// Initialization happens below, via calls to addBuilder.
all: change dashboard.BuildConfig to be a pointer type, like HostConfig BuildConfig grew way too large and has too many slices & such to be a value type. Make it a pointer type, which matches HostConfig. Change-Id: Ie625bece9d6d8c1ec6cff26e77416bd9b1f256d8 Reviewed-on: https://go-review.googlesource.com/c/145077 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 20:52:25 +03:00
var Builders = map[string]*BuildConfig{}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// Hosts contains the names and configs of all the types of
// buildlets. They can be VMs, containers, or dedicated machines.
var Hosts = map[string]*HostConfig{
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
"host-linux-jessie": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Debian Jessie, our standard Linux container image.",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
ContainerImage: "linux-x86-jessie:latest",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
"host-linux-stretch": &HostConfig{
Notes: "Debian Stretch",
ContainerImage: "linux-x86-stretch:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "root",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
"host-linux-stretch-vmx": &HostConfig{
Notes: "Debian Stretch w/ Nested Virtualization (VMX CPU bit) enabled, for testing",
ContainerImage: "linux-x86-stretch:latest",
NestedVirt: true,
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
cmd/coordinator: cross-compile ARM build on Kubernetes Saves 4.5 minutes or so by using fast x86 machines to build the ARM build instead of running make.bash on Scaleway ARM machines. We still run the tests on ARM, and have a separate builder only running make.bash on ARM (see prior golang.org/cl/29670) Fixes golang/go#17105 Updates golang/go#17104 Change-Id: I1cb7b0e5b1cc8b644195f262328884ed3aff120a Reviewed-on: https://go-review.googlesource.com/29677 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-23 23:44:43 +03:00
"host-linux-armhf-cross": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Debian Jessie with armhf cross-compiler, built from env/crosscompile/linux-armhf-jessie",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-armhf-jessie:latest",
cmd/coordinator: cross-compile ARM build on Kubernetes Saves 4.5 minutes or so by using fast x86 machines to build the ARM build instead of running make.bash on Scaleway ARM machines. We still run the tests on ARM, and have a separate builder only running make.bash on ARM (see prior golang.org/cl/29670) Fixes golang/go#17105 Updates golang/go#17104 Change-Id: I1cb7b0e5b1cc8b644195f262328884ed3aff120a Reviewed-on: https://go-review.googlesource.com/29677 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-23 23:44:43 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
cmd/coordinator, dashboard: cross compile linux-arm-arm5spacemonkey Change-Id: I3a228627ec8236d0d10bb9a1a61443c81493145d Reviewed-on: https://go-review.googlesource.com/38460 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-23 18:46:39 +03:00
"host-linux-armel-cross": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Debian Jessie with armel cross-compiler, from env/crosscompile/linux-armel-stretch",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-armel-stretch:latest",
cmd/coordinator, dashboard: cross compile linux-arm-arm5spacemonkey Change-Id: I3a228627ec8236d0d10bb9a1a61443c81493145d Reviewed-on: https://go-review.googlesource.com/38460 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-23 18:46:39 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
"host-linux-amd64-localdev": &HostConfig{
IsReverse: true,
ExpectNum: 0,
Notes: "for localhost development of buildlets/gomote/coordinator only",
SSHUsername: os.Getenv("USER"),
},
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
"host-nacl-arm-davecheney": &HostConfig{
IsReverse: true,
ExpectNum: 1,
Notes: "Raspberry Pi 3",
OwnerGithub: "davecheney",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-nacl-kube": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Container with Native Client binaries.",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-nacl:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
"host-js-wasm": &HostConfig{
Notes: "Container with node.js for testing js/wasm.",
ContainerImage: "js-wasm:latest",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-s390x-cross-kube": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Container with s390x cross-compiler.",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-s390x-stretch:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
env: add alpine linux builder Adds flag to upload to build binary statically. Updates golang/go#17891 Change-Id: If19b2e2cf9a84fa2b4cc6fdf55d8add5b54abff7 Reviewed-on: https://go-review.googlesource.com/33890 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-02 22:14:12 +03:00
"host-linux-x86-alpine": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Alpine container",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-alpine:latest",
env: add alpine linux builder Adds flag to upload to build binary statically. Updates golang/go#17891 Change-Id: If19b2e2cf9a84fa2b4cc6fdf55d8add5b54abff7 Reviewed-on: https://go-review.googlesource.com/33890 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-02 22:14:12 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64-static",
env: []string{"GOROOT_BOOTSTRAP=/usr/lib/go"},
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-linux-clang": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Container with clang.",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-clang:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
"host-linux-sid": &HostConfig{
env/linux-x86-sid: convert from GCE VM to Kubernetes And update it. What we used to call "sid" was probably ancient. Change-Id: I76c1aad60eb655513119eaad0550ba5e0df3e9c5 Reviewed-on: https://go-review.googlesource.com/40399 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 04:54:09 +03:00
Notes: "Debian sid, updated occasionally.",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-sid:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
dashboard: builders.go change for new scaleway docker config Working towards golang/go#20397 Change-Id: I0861f0a265498840cd7e852504cd5b9e53893036 Reviewed-on: https://go-review.googlesource.com/43813 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-22 23:17:48 +03:00
"host-linux-arm-scaleway": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
IsReverse: true,
HermeticReverse: true,
ExpectNum: 50,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
ReverseAliases: []string{"linux-arm", "linux-arm-arm5"},
SSHUsername: "root",
dashboard: builders.go change for new scaleway docker config Working towards golang/go#20397 Change-Id: I0861f0a265498840cd7e852504cd5b9e53893036 Reviewed-on: https://go-review.googlesource.com/43813 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-22 23:17:48 +03:00
},
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
"host-linux-arm5spacemonkey": &HostConfig{
IsReverse: true,
ExpectNum: 3,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
ReverseAliases: []string{"linux-arm-arm5spacemonkey"},
dashboard: remove fake ARM5 builders The spacemonkey actually-ARM5 builders have been working and are better. The arm5 builders being deleted here were Scaleway C1 servers (which have better-than-ARM5 hardware) but running with GOARM=5-compiled binaries. That test a bit, but not enough. Fixes golang/go#19662 Fixes golang/go#19848 (looks like this is only missing owner annotation) Change-Id: I1d3b21b0139d4d4ccf2c0baebd4f2f01b4024861 Reviewed-on: https://go-review.googlesource.com/41618 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-25 02:07:45 +03:00
OwnerGithub: "zeebo",
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
},
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
"host-openbsd-amd64-60": &HostConfig{
VMImage: "openbsd-amd64-60",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-60.tar.gz",
Notes: "OpenBSD 6.0; GCE VM is built from script in build/env/openbsd-amd64",
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
},
dashboard: switch to OpenBSD 6.0 completely, remove OpenBSD 5.8 Updates golang/go#16951 Change-Id: I7b92363671e0ba4c6ab80cef60e7c8600687f440 Reviewed-on: https://go-review.googlesource.com/34384 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-12-15 01:41:25 +03:00
"host-openbsd-386-60": &HostConfig{
VMImage: "openbsd-386-60",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386",
dashboard: switch to OpenBSD 6.0 completely, remove OpenBSD 5.8 Updates golang/go#16951 Change-Id: I7b92363671e0ba4c6ab80cef60e7c8600687f440 Reviewed-on: https://go-review.googlesource.com/34384 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-12-15 01:41:25 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-60.tar.gz",
Notes: "OpenBSD 6.0; GCE VM is built from script in build/env/openbsd-386",
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
"host-openbsd-amd64-62": &HostConfig{
VMImage: "openbsd-amd64-62",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-60.tar.gz",
Notes: "OpenBSD 6.2; GCE VM is built from script in build/env/openbsd-amd64",
SSHUsername: "gopher",
},
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
"host-openbsd-386-62": &HostConfig{
VMImage: "openbsd-386-62-a",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-60.tar.gz",
Notes: "OpenBSD 6.2; GCE VM is built from script in build/env/openbsd-386",
SSHUsername: "gopher",
},
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
"host-openbsd-amd64-64": &HostConfig{
env/openbsd-amd64: force tsc time source, double the build speed Fixes golang/go#29223 Change-Id: I43d1fc1eb6455c063d3842c330cb7d57295d7482 Reviewed-on: https://go-review.googlesource.com/c/160319 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-30 08:01:54 +03:00
VMImage: "openbsd-amd64-64-190129a",
buildlet, dashboard: add min CPU platform knob, require Skylake for OpenBSD I thought this would be enough for OpenBSD to select the TSC on its own without being forced to (as in CL 160319), but apparently it is not: https://github.com/golang/go/issues/29223#issuecomment-459035482 So it seems like we want both this CL and CL 160319. Updates golang/go#29223 Change-Id: I0a092d62881d8dcce0ef1129d8d32d8f4025b6ac Reviewed-on: https://go-review.googlesource.com/c/160457 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-30 20:13:07 +03:00
MinCPUPlatform: "Intel Skylake", // for better TSC? Maybe? see Issue 29223. builds faster at least.
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64-64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-64.tar.gz",
env/openbsd-amd64: enable SMT on OpenBSD 6.4 Disable Spectre/Meltdown mitigations and enable simultaneous multithreading (SMT) on the OpenBSD 6.4 builder image. This was done by setting hw.smt=1 in /etc/sysctl.conf when building the image. I considered an alternative approach of doing this at runtime via a cmd/buildlet special case for OpenBSD. It would've likely been faster to implement if I had the idea at the beginning. However, by the time I saw it, I had already started to build a new OpenBSD 6.4 image with the /etc/sysctl.conf file, then tested it via debugnewvm. At this point, there's no longer a time saving advantage, so I decided to prefer the v2 image because it keeps OpenBSD-specific configuration more contained in env/openbsd-* directory, rather than spreading it out between there and the cmd/buildlet runtime code. It seems to be a slightly better option. The times to do a full build, as measured via cmd/debugnewvm, were: 6.2 = 19m25.218s 6.4 = 28m49.565s 6.4 with hw.smt=1 = 22m55.909s That should translate to faster trybot runs, which is important for open source project health. The 386 environment doesn't need to be updated because it doesn't support hw.smt: sysctl: hw.smt: value is not available Attempting to set it anyway should be harmless and okay to do, in case it happens to get supported in the future. We're only using amd64 environment for trybots, so it's okay for the purposes of golang/go#28403. Reference: https://man.openbsd.org/sysctl.2#HW_SMT_2 Fixes golang/go#28403 Change-Id: I7bc4cbf83ccbdb3aa9dd19eeabd88feb1c425811 Reviewed-on: https://go-review.googlesource.com/c/145022 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 20:05:59 +03:00
Notes: "OpenBSD 6.4 with hw.smt=1; GCE VM is built from script in build/env/openbsd-amd64",
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
SSHUsername: "gopher",
},
"host-openbsd-386-64": &HostConfig{
dashboard: update OpenBSD 6.4 builders to final release Previously, they were running a 6.4 snapshot release from October 4, 2018, added in CL 139857. The final release of OpenBSD 6.4 has been released on October 18, 2018, and can be used now. See https://www.openbsd.org/64.html. Fixup unbound variable in env/openbsd-amd64/make.bash: $ ./make.bash ./make.bash: line 57: PKG_ADD_OPTIONS: unbound variable The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: I0e3b1ed5aa9b2e624d9afbe46a8a344f88ddcc56 Reviewed-on: https://go-review.googlesource.com/c/143458 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-19 22:50:28 +03:00
VMImage: "openbsd-386-64",
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386-64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-64.tar.gz",
dashboard: update OpenBSD 6.4 builders to final release Previously, they were running a 6.4 snapshot release from October 4, 2018, added in CL 139857. The final release of OpenBSD 6.4 has been released on October 18, 2018, and can be used now. See https://www.openbsd.org/64.html. Fixup unbound variable in env/openbsd-amd64/make.bash: $ ./make.bash ./make.bash: line 57: PKG_ADD_OPTIONS: unbound variable The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: I0e3b1ed5aa9b2e624d9afbe46a8a344f88ddcc56 Reviewed-on: https://go-review.googlesource.com/c/143458 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-19 22:50:28 +03:00
Notes: "OpenBSD 6.4; GCE VM is built from script in build/env/openbsd-386",
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
SSHUsername: "gopher",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-freebsd-93-gce": &HostConfig{
VMImage: "freebsd-amd64-gce93",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
"host-freebsd-10_3": &HostConfig{
dashboard: rebuild FreeBSD VM to avoid 10 second boot prompt FreeBSD VMs boot up 10 seconds faster now, and shut down faster. Updates golang/go#22894 Updates golang/go#22895 Change-Id: I8eef979fc43d95d3c03104a761e70ec5737a3a97 Reviewed-on: https://go-review.googlesource.com/80304 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-29 01:07:29 +03:00
VMImage: "freebsd-amd64-103-b",
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
Notes: "FreeBSD 10.3; GCE VM is built from script in build/env/freebsd-amd64",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64", // TODO(bradfitz): why was this http instead of https?
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
env: []string{"CC=clang"},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
"host-freebsd-10_4": &HostConfig{
VMImage: "freebsd-amd64-104",
Notes: "FreeBSD 10.4; GCE VM is built from script in build/env/freebsd-amd64",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
SSHUsername: "gopher",
},
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
"host-freebsd-11_1": &HostConfig{
dashboard: rebuild FreeBSD VM to avoid 10 second boot prompt FreeBSD VMs boot up 10 seconds faster now, and shut down faster. Updates golang/go#22894 Updates golang/go#22895 Change-Id: I8eef979fc43d95d3c03104a761e70ec5737a3a97 Reviewed-on: https://go-review.googlesource.com/80304 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-29 01:07:29 +03:00
VMImage: "freebsd-amd64-111-b",
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
Notes: "FreeBSD 11.1; GCE VM is built from script in build/env/freebsd-amd64",
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64", // TODO(bradfitz): why was this http instead of https?
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
env: []string{"CC=clang"},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
},
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
"host-freebsd-11_2": &HostConfig{
VMImage: "freebsd-amd64-112",
Notes: "FreeBSD 11.2; GCE VM is built from script in build/env/freebsd-amd64",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
SSHUsername: "gopher",
},
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
"host-freebsd-12_0": &HostConfig{
env/freebsd-amd64: update FreeBSD 12.0 to a -RELEASE image Install GDB from the ports collection, the in-tree was removed from the base system in FreeBSD 12.0, and the older releases have an unsupported (6.1.1 or older version). Updates golang/go#25289 Change-Id: I8899a2a70c0b9ee9171621206431ca24a7e97704 Reviewed-on: https://go-review.googlesource.com/c/155977 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-12-29 20:53:25 +03:00
VMImage: "freebsd-amd64-120-v1",
Notes: "FreeBSD 12.0; GCE VM is built from script in build/env/freebsd-amd64",
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
SSHUsername: "gopher",
},
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
"host-netbsd-amd64-8_0": &HostConfig{
VMImage: "netbsd-amd64-8-0-2018q1",
Notes: "NetBSD 8.0RC1; GCE VM is built from script in build/env/netbsd-amd64",
dashboard: bump netbsd-amd64 bootstrap, enlarge VMs, remove 386 GOHOSTARCH Prepared using https://golang.org/cl/80895 Updates golang/go#20852 Change-Id: I186f74010f0a8be66eef5c369be664abe3f68f7f Reviewed-on: https://go-review.googlesource.com/80896 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-30 05:32:26 +03:00
machineType: "n1-highcpu-4",
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.netbsd-amd64",
dashboard: bump netbsd-amd64 bootstrap, enlarge VMs, remove 386 GOHOSTARCH Prepared using https://golang.org/cl/80895 Updates golang/go#20852 Change-Id: I186f74010f0a8be66eef5c369be664abe3f68f7f Reviewed-on: https://go-review.googlesource.com/80896 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-30 05:32:26 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-netbsd-amd64-2da6b33.tar.gz",
dashboard, cmd/buildlet, env/netbsd: enable netbsd ssh, re-disable netbsd-386 Fixes golang/go#22872 Updates golang/go#20852 Change-Id: I2b3454cb59a434cb12b10bb93cb4dd21d5187281 Reviewed-on: https://go-review.googlesource.com/81356 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-12-01 09:55:37 +03:00
SSHUsername: "root",
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
},
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
// Note: the netbsd-386 host hangs during the ../test phase of all.bash,
// so we don't use this for now. (See the netbsd-386-8 BuildConfig below.)
"host-netbsd-386-8_0": &HostConfig{
VMImage: "netbsd-386-8-0-2018q1",
Notes: "NetBSD 8.0RC1; GCE VM is built from script in build/env/netbsd-386",
dashboard: bump netbsd-amd64 bootstrap, enlarge VMs, remove 386 GOHOSTARCH Prepared using https://golang.org/cl/80895 Updates golang/go#20852 Change-Id: I186f74010f0a8be66eef5c369be664abe3f68f7f Reviewed-on: https://go-review.googlesource.com/80896 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-30 05:32:26 +03:00
machineType: "n1-highcpu-4",
all: start to revive netbsd-386 support It's still busted, but this will let other people debug. Updates golang/go#22914 Updates golang/go#20852 Updates golang/go#19339 Change-Id: Iedc21417e56418dab6abd433574ffef012ae43f3 Reviewed-on: https://go-review.googlesource.com/80435 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-29 03:44:46 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.netbsd-386",
dashboard: re-disable netbsd/386 builder until binary packages are built Also update the bootstrap toolchain (verified in VMware VM locally). Updates golang/go#20852 Change-Id: Iac875419ed8dc51fb9db09d8948067f24c601e17 Reviewed-on: https://go-review.googlesource.com/82018 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-12-06 02:39:30 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-netbsd-386-0b3b511.tar.gz",
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
SSHUsername: "root",
all: start to revive netbsd-386 support It's still busted, but this will let other people debug. Updates golang/go#22914 Updates golang/go#20852 Updates golang/go#19339 Change-Id: Iedc21417e56418dab6abd433574ffef012ae43f3 Reviewed-on: https://go-review.googlesource.com/80435 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-29 03:44:46 +03:00
},
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
"host-dragonfly-amd64-tdfbsd": &HostConfig{
IsReverse: true,
ExpectNum: 1,
dashboard: set GOROOT_BOOTSTRAP for dragonfly-amd64 and freebsd-arm Updates golang/go#21191 Change-Id: I09854a99c17dec204ee3a1f7c197bb7b69895979 Reviewed-on: https://go-review.googlesource.com/53491 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 02:14:51 +03:00
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
ReverseAliases: []string{"dragonfly-amd64"},
OwnerGithub: "tdfbsd",
},
"host-freebsd-arm-paulzhol": &HostConfig{
IsReverse: true,
ExpectNum: 1,
dashboard: add freebsd-arm host notes Copied from https://github.com/golang/go/wiki/DashboardBuilders Change-Id: If69e5fc85af380bf92f408c68b481d0cd5cc1bdb Reviewed-on: https://go-review.googlesource.com/53359 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 06:57:03 +03:00
Notes: "Cubiboard2 1Gb RAM dual-core Cortex-A7 (Allwinner A20), FreeBSD 11.1-RELEASE",
dashboard: set GOROOT_BOOTSTRAP for dragonfly-amd64 and freebsd-arm Updates golang/go#21191 Change-Id: I09854a99c17dec204ee3a1f7c197bb7b69895979 Reviewed-on: https://go-review.googlesource.com/53491 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 02:14:51 +03:00
env: []string{"GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4"},
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
ReverseAliases: []string{"freebsd-arm-paulzhol"},
OwnerGithub: "paulzhol",
},
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
"host-plan9-arm-0intro": &HostConfig{
IsReverse: true,
ExpectNum: 1,
Notes: "Raspberry Pi 3 Model B, Plan 9 from Bell Labs",
OwnerGithub: "0intro",
},
"host-plan9-amd64-0intro": &HostConfig{
IsReverse: true,
ExpectNum: 1,
OwnerGithub: "0intro",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-plan9-386-gce": &HostConfig{
env/plan9: revert SMP changes, stop running cmd/go tests Revert "env: enable SMP in Plan 9 image" This reverts commit 6478d3e7bdff4c8d810ba44f970ca4b0bfc92405. Revert "dashboard: bump plan9 GCE image" This reverts commit ce208996bf954a87d68789bb6a9000256f016c77. Updates golang/go#29822 Change-Id: I47e75d135e092ee9d5a80c66e91c1048c67cd5f6 Reviewed-on: https://go-review.googlesource.com/c/158542 Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: David du Colombier <0intro@gmail.com>
2019-01-19 01:19:58 +03:00
VMImage: "plan9-386-v7",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "Plan 9 from 0intro; GCE VM is built from script in build/env/plan9-386",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.plan9-386",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-plan9-386.tar.gz",
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// We *were* using n1-standard-1 because Plan 9 can only
// reliably use a single CPU. Using 2 or 4 and we see
// test failures. See:
// https://golang.org/issue/8393
// https://golang.org/issue/9491
// n1-standard-1 has 3.6 GB of memory which WAS (see below)
// overkill (userspace probably only sees 2GB anyway),
// but it's the cheapest option. And plenty to keep
// our ~250 MB of inputs+outputs in its ramfs.
//
// But the docs says "For the n1 series of machine
// types, a virtual CPU is implemented as a single
// hyperthread on a 2.6GHz Intel Sandy Bridge Xeon or
// Intel Ivy Bridge Xeon (or newer) processor. This
// means that the n1-standard-2 machine type will see
// a whole physical core."
//
// ... so we used n1-highcpu-2 (1.80 RAM, still
// plenty), just so we can get 1 whole core for the
// single-core Plan 9. It will see 2 virtual cores and
// only use 1, but we hope that 1 will be more powerful
// and we'll stop timing out on tests.
machineType: "n1-highcpu-4",
dashboard: increase plan9's GO_TEST_TIMEOUT_SCALE Updates golang/go#29801 Change-Id: Ib346d717ae9b8de18666d6ea543d9bdcd2856f3b Reviewed-on: https://go-review.googlesource.com/c/158537 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-01-18 19:14:12 +03:00
env: []string{"GO_TEST_TIMEOUT_SCALE=3"},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
"host-windows-amd64-2008": &HostConfig{
env/windows: disable windows defender on builders Windows Defender scans all new files written to disk which slows down build performance. - Uninstall Windows Defender as part of image building - Refactor/Cleanup registry section for consistency - Promote new image version to dashboard (built in staging/prod) fixes golang/go#26055 Change-Id: I3e16b9a9581824c20abed5e8ffae1efd46c6dd09 Reviewed-on: https://go-review.googlesource.com/121937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-02 20:49:55 +03:00
VMImage: "windows-amd64-server-2008r2-v7",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
dashboard,cmd/buildlet,cmd/coordinator: public key auth on windows Enable public key auth for windows buildlets. Previously the username/password was hardcoded. This change enables the `/connect-ssh` http endpoint on the buildlet to setup authorized_keys and uses the coordinator to proxy ssh traffic to the buildlet. Validation: - run local coordinator (with patches to dev env to act like GCE) with changes - build windows buildlet, post to own bucket - execute: gomote -localdev create windows-amd64-2012 ssh -p 2222 user-jrjohnson-windows-amd64-2012-0@127.0.0.1 Fixes golang/go#21247 Change-Id: If7adc7e6296a071d50412d9fa9d46ecd9e6cc3f3 Reviewed-on: https://go-review.googlesource.com/52350 Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-08-01 02:55:28 +03:00
SSHUsername: "gopher",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
},
"host-windows-amd64-2012": &HostConfig{
env/windows: disable windows defender on builders Windows Defender scans all new files written to disk which slows down build performance. - Uninstall Windows Defender as part of image building - Refactor/Cleanup registry section for consistency - Promote new image version to dashboard (built in staging/prod) fixes golang/go#26055 Change-Id: I3e16b9a9581824c20abed5e8ffae1efd46c6dd09 Reviewed-on: https://go-review.googlesource.com/121937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-02 20:49:55 +03:00
VMImage: "windows-amd64-server-2012r2-v7",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
dashboard,cmd/buildlet,cmd/coordinator: public key auth on windows Enable public key auth for windows buildlets. Previously the username/password was hardcoded. This change enables the `/connect-ssh` http endpoint on the buildlet to setup authorized_keys and uses the coordinator to proxy ssh traffic to the buildlet. Validation: - run local coordinator (with patches to dev env to act like GCE) with changes - build windows buildlet, post to own bucket - execute: gomote -localdev create windows-amd64-2012 ssh -p 2222 user-jrjohnson-windows-amd64-2012-0@127.0.0.1 Fixes golang/go#21247 Change-Id: If7adc7e6296a071d50412d9fa9d46ecd9e6cc3f3 Reviewed-on: https://go-review.googlesource.com/52350 Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-08-01 02:55:28 +03:00
SSHUsername: "gopher",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
},
"host-windows-amd64-2016": &HostConfig{
env/windows: disable windows defender on builders Windows Defender scans all new files written to disk which slows down build performance. - Uninstall Windows Defender as part of image building - Refactor/Cleanup registry section for consistency - Promote new image version to dashboard (built in staging/prod) fixes golang/go#26055 Change-Id: I3e16b9a9581824c20abed5e8ffae1efd46c6dd09 Reviewed-on: https://go-review.googlesource.com/121937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-02 20:49:55 +03:00
VMImage: "windows-amd64-server-2016-v7",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
dashboard,cmd/buildlet,cmd/coordinator: public key auth on windows Enable public key auth for windows buildlets. Previously the username/password was hardcoded. This change enables the `/connect-ssh` http endpoint on the buildlet to setup authorized_keys and uses the coordinator to proxy ssh traffic to the buildlet. Validation: - run local coordinator (with patches to dev env to act like GCE) with changes - build windows buildlet, post to own bucket - execute: gomote -localdev create windows-amd64-2012 ssh -p 2222 user-jrjohnson-windows-amd64-2012-0@127.0.0.1 Fixes golang/go#21247 Change-Id: If7adc7e6296a071d50412d9fa9d46ecd9e6cc3f3 Reviewed-on: https://go-review.googlesource.com/52350 Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-08-01 02:55:28 +03:00
SSHUsername: "gopher",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
},
dashboard: add windows-arm builder Updates golang/go#26148 Change-Id: I23a2c7bd1c2f8f7c960aaedfe0dcdee4e11825ab Reviewed-on: https://go-review.googlesource.com/125643 Reviewed-by: Andrew Bonventre <andybons@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-24 22:59:34 +03:00
"host-windows-arm-iotcore": &HostConfig{
IsReverse: true,
ExpectNum: 1,
OwnerGithub: "jordanrh1",
env: []string{"GOROOT_BOOTSTRAP=C:\\Data\\Go"},
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-darwin-10_8": &HostConfig{
IsReverse: true,
dashboard: adjust how many Mac VMs we expect Updates golang/go#23859 Change-Id: I7e0fed5b17430669a726a8a7bbf8d3efc190034c Reviewed-on: https://go-review.googlesource.com/95735 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-02-21 04:25:11 +03:00
ExpectNum: 0,
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "MacStadium OS X 10.8 VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ReverseAliases: []string{"darwin-amd64-10_8"},
SSHUsername: "gopher",
HermeticReverse: false, // TODO: make it so, like 10.12
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
"host-darwin-10_10": &HostConfig{
IsReverse: true,
dashboard: adjust how many Mac VMs we expect Updates golang/go#23859 Change-Id: I7e0fed5b17430669a726a8a7bbf8d3efc190034c Reviewed-on: https://go-review.googlesource.com/95735 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-02-21 04:25:11 +03:00
ExpectNum: 1,
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "MacStadium OS X 10.10 VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ReverseAliases: []string{"darwin-amd64-10_10"},
SSHUsername: "gopher",
HermeticReverse: false, // TODO: make it so, like 10.12
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
"host-darwin-10_11": &HostConfig{
IsReverse: true,
dashboard: adjust how many Mac VMs we expect Updates golang/go#23859 Change-Id: I7e0fed5b17430669a726a8a7bbf8d3efc190034c Reviewed-on: https://go-review.googlesource.com/95735 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-02-21 04:25:11 +03:00
ExpectNum: 17,
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "MacStadium OS X 10.11 VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ReverseAliases: []string{"darwin-amd64-10_11"},
SSHUsername: "gopher",
HermeticReverse: false, // TODO: make it so, like 10.12
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
"host-darwin-10_12": &HostConfig{
IsReverse: true,
cmd/coordinator: clean up reverse buildlet code, export status JSON Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-24 20:50:49 +03:00
ExpectNum: 2,
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
Notes: "MacStadium OS X 10.12 VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ReverseAliases: []string{"darwin-amd64-10_12"},
SSHUsername: "gopher",
HermeticReverse: true, // we destroy the VM when done & let cmd/makemac recreate
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-linux-s390x": &HostConfig{
Notes: "run by IBM",
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
OwnerGithub: "mundaym",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
IsReverse: true,
env: []string{"GOROOT_BOOTSTRAP=/var/buildlet/go-linux-s390x-bootstrap"},
ReverseAliases: []string{"linux-s390x-ibm"},
},
all: add ppc64 (big endian) builders Change-Id: I0fddda33179555a50b6f03dcff13338544b742c3 Reviewed-on: https://go-review.googlesource.com/30599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-06 23:30:58 +03:00
"host-linux-ppc64-osu": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
Notes: "Debian jessie; run by Go team on osuosl.org",
IsReverse: true,
ExpectNum: 5,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
ReverseAliases: []string{"linux-ppc64-buildlet"},
SSHUsername: "debian",
HermeticReverse: false, // TODO: use rundockerbuildlet like arm64
all: add ppc64 (big endian) builders Change-Id: I0fddda33179555a50b6f03dcff13338544b742c3 Reviewed-on: https://go-review.googlesource.com/30599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-06 23:30:58 +03:00
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-linux-ppc64le-osu": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
Notes: "Debian jessie; run by Go team on osuosl.org",
IsReverse: true,
ExpectNum: 5,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
ReverseAliases: []string{"linux-ppc64le-buildlet"},
SSHUsername: "debian",
HermeticReverse: false, // TODO: use rundockerbuildlet like arm64
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard: add linux-ppc64le POWER9 builder This change adds an extra linux-ppc64le builder running on POWER9. Fixes golang/go#30138 Change-Id: I23bdfc116bb89c6d96be330c2c87827c47705b6b Reviewed-on: https://go-review.googlesource.com/c/163757 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:05:12 +03:00
"host-linux-ppc64le-power9-osu": &HostConfig{
Notes: "Debian jessie; run by IBM",
OwnerGithub: "ceseo",
IsReverse: true,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
ReverseAliases: []string{"linux-ppc64le-power9osu"},
SSHUsername: "debian",
HermeticReverse: false, // TODO: use rundockerbuildlet like arm64
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-linux-arm64-linaro": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
Notes: "Ubuntu xenial; run by Go team, from linaro",
IsReverse: true,
HermeticReverse: true,
ExpectNum: 5,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
ReverseAliases: []string{"linux-arm64-buildlet"},
SSHUsername: "root",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
"host-linux-arm64-packet": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
Notes: "On 96 core packet.net host (Xenial) in Docker containers (Jessie); run by Go team. See x/build/env/linux-arm64/packet",
IsReverse: true,
HermeticReverse: true,
ExpectNum: 20,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
SSHUsername: "root",
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-solaris-amd64": &HostConfig{
Notes: "run by Go team on Joyent, on a SmartOS 'infrastructure container'",
IsReverse: true,
cmd/coordinator: clean up reverse buildlet code, export status JSON Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-24 20:50:49 +03:00
ExpectNum: 5,
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
env: []string{"GOROOT_BOOTSTRAP=/root/go-solaris-amd64-bootstrap"},
ReverseAliases: []string{"solaris-amd64-smartosbuildlet"},
},
x/build: add Solaris amd64 release builder to dashboard Fixes golang/go#20949 Change-Id: I4f85ee76783a1240a2ebd4a2a1f167ab849fcebe Reviewed-on: https://go-review.googlesource.com/47811 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-07 23:41:20 +03:00
"host-solaris-oracle-amd64-oraclerel": &HostConfig{
Notes: "Oracle Solaris amd64 Release System",
Owner: "shawn.walker@oracle.com",
OwnerGithub: "binarycrusader",
IsReverse: true,
ExpectNum: 1,
x/build: bootstrap tweaks for solaris builders Change bootstrap logic for solaris builders to avoid applying it to Oracle Solaris builders. Additionally, update the location of the Go bootstrap to reflect actual Oracle Solaris builder configuration. Fixes golang/go#21241 Change-Id: Ia2f21fab1581b4c195256c0dca7a87b4212798bf Reviewed-on: https://go-review.googlesource.com/52310 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-31 23:58:19 +03:00
env: []string{"GOROOT_BOOTSTRAP=/opt/golang/go-solaris-amd64-bootstrap"},
x/build: add Solaris amd64 release builder to dashboard Fixes golang/go#20949 Change-Id: I4f85ee76783a1240a2ebd4a2a1f167ab849fcebe Reviewed-on: https://go-review.googlesource.com/47811 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-07 23:41:20 +03:00
},
dashboard: add Oracle Solaris development builder Fixes golang/go#15072 Change-Id: I85677636a0f7f8b6a7b2405bb0669c39cc1da7f0 Reviewed-on: https://go-review.googlesource.com/46831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-27 20:11:15 +03:00
"host-solaris-oracle-shawn": &HostConfig{
Notes: "Oracle Solaris amd64 Development System",
Owner: "shawn.walker@oracle.com",
OwnerGithub: "binarycrusader",
IsReverse: true,
ExpectNum: 1,
x/build: bootstrap tweaks for solaris builders Change bootstrap logic for solaris builders to avoid applying it to Oracle Solaris builders. Additionally, update the location of the Go bootstrap to reflect actual Oracle Solaris builder configuration. Fixes golang/go#21241 Change-Id: Ia2f21fab1581b4c195256c0dca7a87b4212798bf Reviewed-on: https://go-review.googlesource.com/52310 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-31 23:58:19 +03:00
env: []string{"GOROOT_BOOTSTRAP=/opt/golang/go-solaris-amd64-bootstrap"},
dashboard: add Oracle Solaris development builder Fixes golang/go#15072 Change-Id: I85677636a0f7f8b6a7b2405bb0669c39cc1da7f0 Reviewed-on: https://go-review.googlesource.com/46831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-27 20:11:15 +03:00
},
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
"host-linux-mips": &HostConfig{
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
Notes: "Run by Brendan Kirby, imgtec.com",
OwnerGithub: "MIPSbkirby",
IsReverse: true,
ExpectNum: 1,
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mips",
"GOARCH=mips",
"GOHOSTARCH=mips",
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
"GO_TEST_TIMEOUT_SCALE=4",
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
},
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
ReverseAliases: []string{"linux-mips"},
},
"host-linux-mipsle": &HostConfig{
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
Notes: "Run by Brendan Kirby, imgtec.com",
OwnerGithub: "MIPSbkirby",
IsReverse: true,
ExpectNum: 1,
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mipsle",
"GOARCH=mipsle",
"GOHOSTARCH=mipsle",
},
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
ReverseAliases: []string{"linux-mipsle"},
},
"host-linux-mips64": &HostConfig{
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
Notes: "Run by Brendan Kirby, imgtec.com",
OwnerGithub: "MIPSbkirby",
IsReverse: true,
ExpectNum: 1,
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mips64",
"GOARCH=mips64",
"GOHOSTARCH=mips64",
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
"GO_TEST_TIMEOUT_SCALE=4",
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
},
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
ReverseAliases: []string{"linux-mips64"},
},
"host-linux-mips64le": &HostConfig{
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
Notes: "Run by Brendan Kirby, imgtec.com",
OwnerGithub: "MIPSbkirby",
IsReverse: true,
ExpectNum: 1,
dashboard: set GOARCH and GOHOSTARCH explicitly for mips builders They're running the 32-bit versions on 64-bit hardware. Change-Id: I365aa2b77cd729484d49aaefed12737c657abed8 Reviewed-on: https://go-review.googlesource.com/33010 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2016-11-09 21:53:36 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mips64le",
"GOARCH=mips64le",
"GOHOSTARCH=mips64le",
},
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
ReverseAliases: []string{"linux-mips64le"},
},
dashboard: split mobile builders into iOS and Android Even though both mobile builders will run off the same host Mac Mini, Android and iOS builds can run simultaneously. Split the eliasnaur host in two to allow that. Change-Id: If3db4dadb56b8a4b212dcbb3015e3e6c1b69608d Reviewed-on: https://go-review.googlesource.com/39130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-31 11:08:01 +03:00
"host-darwin-amd64-eliasnaur-android": &HostConfig{
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
Notes: "Mac Mini hosted by Elias Naur, running the android reverse buildlet",
OwnerGithub: "eliasnaur",
IsReverse: true,
ExpectNum: 1,
dashboard: split mobile builders into iOS and Android Even though both mobile builders will run off the same host Mac Mini, Android and iOS builds can run simultaneously. Split the eliasnaur host in two to allow that. Change-Id: If3db4dadb56b8a4b212dcbb3015e3e6c1b69608d Reviewed-on: https://go-review.googlesource.com/39130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-31 11:08:01 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap",
"GOHOSTARCH=amd64",
"GOOS=android",
},
},
dashboard: undo ios builder split Running multiple iOS devices on a single host was too ambitious for the brittle test harness. This CL reverts most of the split from CL 57297 except the device model names are updated to reflect reality. I'm sorry. Change-Id: I4e9b1d677eb7c57178f112dedb62580fc74a461d Reviewed-on: https://go-review.googlesource.com/59510 Reviewed-by: Sarah Adams <shadams@google.com>
2017-08-28 20:11:57 +03:00
"host-darwin-amd64-eliasnaur-ios": &HostConfig{
Notes: "Mac Mini hosted by Elias Naur, running the ios reverse buildlet",
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
OwnerGithub: "eliasnaur",
IsReverse: true,
ExpectNum: 1,
dashboard: convert new mobile builder to new split host/build configs To allow multiple build configs (e.g. darwin/arm and darwin/arm64) on the same builder host, the host must be added to the Hosts map and its reverse builders registered to it through HostType. Change-Id: I0608ffb183f3107a039e963c06f6928606d2cf08 Reviewed-on: https://go-review.googlesource.com/38830 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:04:40 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap",
"GOHOSTARCH=amd64",
},
},
dashboard: add zenly as builder for ios (Iphone 5C, 7+) Change-Id: Ie1dd5c53ed0b2d207297ee0cd3ba46a111167e30 Reviewed-on: https://go-review.googlesource.com/125815 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-25 16:14:32 +03:00
"host-darwin-amd64-zenly-ios": &HostConfig{
Notes: "MacBook Pro hosted by Zenly, running the ios reverse buildlet",
OwnerGithub: "znly",
IsReverse: true,
ExpectNum: 1,
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/Cellar/1.10.3/libexec",
"GOHOSTARCH=amd64",
},
},
dashboard: add aix-ppc64 builder This is a starting point for adding an aix-ppc64 reverse builder. There can be further adjustments made to it as needed. Updates golang/go#27160. Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508 Reviewed-on: https://go-review.googlesource.com/131335 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-24 21:41:05 +03:00
"host-aix-ppc64-osuosl": &HostConfig{
Notes: "AIX 7.2 VM on OSU; run by Tony Reix",
OwnerGithub: "trex58",
IsReverse: true,
ExpectNum: 1,
env: []string{"GOROOT_BOOTSTRAP=/opt/freeware/lib/golang"},
},
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
"host-android-amd64-emu": &HostConfig{
Notes: "Debian Buster w/ Android SDK + emulator (use nested virt)",
env/android-amd64-emu: enable gomobile device tests By adding gradle and Android build tools to PATH, setting ANDROID_HOME, and adding the OpenAL headers to CGO_CFLAGS. Change-Id: Ifeaa1b82f4108afc3787eee4449b3317185bbaec Reviewed-on: https://go-review.googlesource.com/c/build/+/164658 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-01 13:21:20 +03:00
ContainerImage: "android-amd64-emu:bff27c0c9263",
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
KonletVMImage: "android-amd64-emu",
NestedVirt: true,
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
}
func init() {
for key, c := range Hosts {
if key == "" {
panic("empty string key in Hosts")
}
if c.HostType == "" {
c.HostType = key
}
if c.HostType != key {
panic(fmt.Sprintf("HostType %q != key %q", c.HostType, key))
}
nSet := 0
if c.VMImage != "" {
nSet++
}
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
if c.ContainerImage != "" {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
nSet++
}
if c.IsReverse {
nSet++
}
if nSet != 1 {
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
panic(fmt.Sprintf("exactly one of VMImage, ContainerImage, IsReverse must be set for host %q; got %v", key, nSet))
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
}
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
if c.buildletURLTmpl == "" && (c.VMImage != "" || c.ContainerImage != "") {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
panic(fmt.Sprintf("missing buildletURLTmpl for host type %q", key))
}
}
}
// A HostConfig describes the available ways to obtain buildlets of
// different types. Some host configs can server multiple
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
// builders. For example, a host config of "host-linux-jessie" can
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// serve linux-amd64, linux-amd64-race, linux-386, linux-386-387, etc.
type HostConfig struct {
// HostType is the unique name of this host config. It is also
// the key in the Hosts map.
HostType string
// buildletURLTmpl is the URL "template" ($BUCKET is auto-expanded)
// for the URL to the buildlet binary.
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
// This field is required for VM and Container builders. It's not
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// needed for reverse buildlets because in that case, the buildlets
// are already running and their stage0 should know how to update it
// it automatically.
buildletURLTmpl string
// Exactly 1 of these must be set:
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
VMImage string // e.g. "openbsd-amd64-60"
ContainerImage string // e.g. "linux-buildlet-std:latest" (suffix after "gcr.io/<PROJ>/")
IsReverse bool // if true, only use the reverse buildlet pool
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// GCE options, if VMImage != ""
buildlet, dashboard: add min CPU platform knob, require Skylake for OpenBSD I thought this would be enough for OpenBSD to select the TSC on its own without being forced to (as in CL 160319), but apparently it is not: https://github.com/golang/go/issues/29223#issuecomment-459035482 So it seems like we want both this CL and CL 160319. Updates golang/go#29223 Change-Id: I0a092d62881d8dcce0ef1129d8d32d8f4025b6ac Reviewed-on: https://go-review.googlesource.com/c/160457 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-30 20:13:07 +03:00
machineType string // optional GCE instance type
RegularDisk bool // if true, use spinning disk instead of SSD
MinCPUPlatform string // optional; https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
cmd/coordinator: clean up reverse buildlet code, export status JSON Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-24 20:50:49 +03:00
// ReverseOptions:
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ExpectNum int // expected number of reverse buildlets of this type
HermeticReverse bool // whether reverse buildlet has fresh env per conn
cmd/coordinator: clean up reverse buildlet code, export status JSON Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-24 20:50:49 +03:00
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
// Container image options, if ContainerImage != "":
env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx This adds scripts to create a new builder host image that acts like Container-Optimized OS (has docker, runs konlet on startup) but with a Debian 9 kernel + userspace that permits KVM for nested virtualization. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884 Reviewed-on: https://go-review.googlesource.com/c/163301 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-22 00:10:54 +03:00
NestedVirt bool // container requires VMX nested virtualization
KonletVMImage string // optional VM image (containing konlet) to use instead of default
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// Optional base env. GOROOT_BOOTSTRAP should go here if the buildlet
// has Go 1.4+ baked in somewhere.
env []string
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
// These template URLs may contain $BUCKET which is expanded to the
// relevant Cloud Storage bucket as specified by the build environment.
goBootstrapURLTmpl string // optional URL to a built Go 1.4+ tar.gz
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
Owner string // optional email of owner; "bradfitz@golang.org", empty means golang-dev
OwnerGithub string // optional GitHub username of owner
Notes string // notes for humans
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername string // username to ssh as, empty means not supported
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// ReverseAliases lists alternate names for this buildlet
// config, for older clients doing a reverse dial into the
// coordinator from outside. This prevents us from updating
// 75+ dedicated machines/VMs atomically, switching them to
// the new "host-*" names.
// This is only applicable if IsReverse.
ReverseAliases []string
}
// A BuildConfig describes how to run a builder.
type BuildConfig struct {
// Name is the unique name of the builder, in the form of
// "GOOS-GOARCH" or "GOOS-GOARCH-suffix". For example,
// "darwin-386", "linux-386-387", "linux-amd64-race". Some
// suffixes are well-known and carry special meaning, such as
// "-race".
Name string
// HostType is the required key into the Hosts map, describing
// the type of host this build will run on.
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
// For example, "host-linux-jessie".
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
HostType string
Notes string // notes for humans
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot func(proj string) bool // if non-nil, policy func for whether trybots enabled. nil means off.
TryOnly bool // only used for trybots, and not regular builds
CompileOnly bool // if true, compile tests, but don't run them
FlakyNet bool // network tests are flaky (try anyway, but ignore some failures)
coordinator: rate-limit GCE calls, put cap on number of VMs at once Also, make Windows use regular disks for now, since its image is so large (100 GB) and we only have 2TB of SSD quota. This is all very conservative and paranoid for now until I figure out what part of the coordinator was misbehaving. Change-Id: Icead5c07cf706c2cfc4d1dd66a108649429018ac Reviewed-on: https://go-review.googlesource.com/7910 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2015-03-21 02:14:52 +03:00
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
// PostSubmitBuildPolicy optionally specifies whether this
// builder does post-submit builds for the given repo ("go",
// "net", etc). If nil, a default policy is used.
PostSubmitBuildPolicy func(repo string) bool
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
// MinimumGoVersion optionally specifies the minimum Go version
// this builder is allowed to use. It can be useful for skipping
// builders that are too new and no longer support some supported
// Go versions. It doesn't need to be set for builders that support
// all supported Go versions.
//
// Note: This field currently has effect on trybot runs only.
MinimumGoVersion types.MajorMinor
cmd/coordinator: support limiting number of builds at once Change-Id: Icd0054f2b943abf2e8a15c32d358ff06541f75f7 Reviewed-on: https://go-review.googlesource.com/41130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-19 23:45:24 +03:00
// MaxAtOnce optionally specifies a cap of how many builds of
// this type can run at once. Zero means unlimited. This is a
// temporary measure until the build scheduler
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
// (golang.org/issue/19178) is done, at which point this field
// should be deleted.
cmd/coordinator: support limiting number of builds at once Change-Id: Icd0054f2b943abf2e8a15c32d358ff06541f75f7 Reviewed-on: https://go-review.googlesource.com/41130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-19 23:45:24 +03:00
MaxAtOnce int
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
// SkipSnapshot, if true, means to not fetch a tarball
// snapshot of the world post-make.bash from the buildlet (and
// thus to not write it to Google Cloud Storage). This is
// incompatible with sharded tests, and should only be used
// for very slow builders or networks, unable to transfer
// the tarball in under ~5 minutes.
SkipSnapshot bool
cmd/coordinator: run benchmarks on try work Benchmarks are treated as unit tests and distributed to the test helpers, which allows them to fit in our 5m trybot budget. Currently we only run the go1 and x/benchmarks. Running package benchmarks is a TODO. This feature is disabled by default, and is enabled by the "farmer-run-bench" project attribute. Updates golang/go#19178 Updates golang/go#19871 Change-Id: I9c3a14da60c3662e7e2cb4e71953060915cc4364 Reviewed-on: https://go-review.googlesource.com/38306 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-15 23:18:30 +03:00
// RunBench causes the coordinator to run benchmarks on this buildlet type.
RunBench bool
dashboard: add a make.bash-only builder on real ARM hardware This is a new builder in prep for the change to the "linux-arm" builder where the GOARCH=arm make.bash will be cross-compiled from a Kubernetes container on fast hardware. Updates golang/go#17105 (cross-compile ARM builders' make.bash) Updates golang/go#17104 (5 minute trybots) Change-Id: Icfd2644d77639f731151abe54839322960418254 Reviewed-on: https://go-review.googlesource.com/29670 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 02:29:31 +03:00
// StopAfterMake causes the build to stop after the make
// script completes, returning its result as the result of the
// whole build. It does not run or compile any of the tests,
// nor does it write a snapshot of the world to cloud
// storage. This option is only supported for builders whose
// BuildConfig.SplitMakeRun returns true.
StopAfterMake bool
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
// InstallRacePackages controls which packages to "go install
// -race <pkgs>" after running make.bash (or equivalent). If
// the builder ends in "-race", the default if non-nil is just
// "std".
InstallRacePackages []string
cmd/coordinator: move git ancestor checking before buildlet creation, generalize This also adds GoDeps to the builders config file and adds a dependency for the ssacheck builder, which currently fails on older release branches with: > Failed on linux-amd64-ssacheck: > https://storage.googleapis.com/go-build-log/1255edaf/linux-amd64-ssacheck_18f7e015.log > > ... > compile: unknown debug key -d dclstack Change-Id: I4ad0a298705feed56f467f3c8d9488f8be2e4af5 Reviewed-on: https://go-review.googlesource.com/43774 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-20 00:23:14 +03:00
// GoDeps is a list of of git sha1 commits that must be in the
// commit to be tested's history. If absent, this builder is
// not run for that commit.
GoDeps []string
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
// ShouldRunDistTest optionally specifies a function which
// controls whether a test (a name from "go tool dist test
// -list") is run. The isTry value is true for trybot runs.
// A few general special cases are handled in
// cmd/coordinator's in buildStatus.shouldSkipTest.
ShouldRunDistTest func(distTestName string, isTry bool) bool
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
// numTestHelpers is the number of _additional_ buildlets
// past the first one to help out with sharded tests.
// For trybots, the numTryHelpers value is used, unless it's
// zero, in which case numTestHelpers is used.
numTestHelpers int
numTryTestHelpers int // for trybots. if 0, numTesthelpers is used
cmd/coordinator: test sharding, better status, logs Also gomote updates which came about during the process of developing and debugging this. Change-Id: Ia53d674118a6b99bcdda7062d3b7161279b6ad52 Reviewed-on: https://go-review.googlesource.com/10463 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-28 07:51:25 +03:00
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
env []string // extra environment ("key=value") pairs
allScriptArgs []string
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
testHostConf *HostConfig // override HostConfig for testing, at least for now
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
}
cmd/coordinator: set GO_BUILDER_NAME env var when exec'ing on buildlet This will allow tests to identify that they are running on the continuous build infrastructure and run extra and/or different tests. Fixes golang/go#11346 Change-Id: Iafe25babc764b7e012c05f0b62331b11690bf194 Reviewed-on: https://go-review.googlesource.com/13120 Reviewed-by: Russ Cox <rsc@golang.org>
2015-08-04 05:19:02 +03:00
func (c *BuildConfig) Env() []string {
dashboard: mark some builders as having flaky networks Follow-up to https://golang.org/cl/22852 Change-Id: I5f0cd25248a8892b7d1b0f14e87b12b8b9a8da3a Reviewed-on: https://go-review.googlesource.com/22883 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2016-05-06 21:55:26 +03:00
env := []string{"GO_BUILDER_NAME=" + c.Name}
if c.FlakyNet {
env = append(env, "GO_BUILDER_FLAKY_NET=1")
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
env = append(env, c.hostConf().env...)
dashboard: mark some builders as having flaky networks Follow-up to https://golang.org/cl/22852 Change-Id: I5f0cd25248a8892b7d1b0f14e87b12b8b9a8da3a Reviewed-on: https://go-review.googlesource.com/22883 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2016-05-06 21:55:26 +03:00
return append(env, c.env...)
cmd/coordinator: set GO_BUILDER_NAME env var when exec'ing on buildlet This will allow tests to identify that they are running on the continuous build infrastructure and run extra and/or different tests. Fixes golang/go#11346 Change-Id: Iafe25babc764b7e012c05f0b62331b11690bf194 Reviewed-on: https://go-review.googlesource.com/13120 Reviewed-by: Russ Cox <rsc@golang.org>
2015-08-04 05:19:02 +03:00
}
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *BuildConfig) IsReverse() bool { return c.hostConf().IsReverse }
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
func (c *BuildConfig) IsContainer() bool { return c.hostConf().IsContainer() }
func (c *HostConfig) IsContainer() bool { return c.ContainerImage != "" }
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
func (c *BuildConfig) IsVM() bool { return c.hostConf().IsVM() }
func (c *HostConfig) IsVM() bool { return c.VMImage != "" }
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
dashboard: create buildlet client package, move coordinator code into it Operation Packification, step 2 of tons. Eventually the buildlet client binary will use this stuff now. Change-Id: I4cf5f3e6beb9e56bdc795ed513ce6daaf61425e3 Reviewed-on: https://go-review.googlesource.com/2921 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 03:29:16 +03:00
func (c *BuildConfig) GOOS() string { return c.Name[:strings.Index(c.Name, "-")] }
func (c *BuildConfig) GOARCH() string {
arch := c.Name[strings.Index(c.Name, "-")+1:]
i := strings.Index(arch, "-")
if i == -1 {
return arch
}
return arch[:i]
}
cmd/coordinator: test sharding, better status, logs Also gomote updates which came about during the process of developing and debugging this. Change-Id: Ia53d674118a6b99bcdda7062d3b7161279b6ad52 Reviewed-on: https://go-review.googlesource.com/10463 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-28 07:51:25 +03:00
// FilePathJoin is mostly like filepath.Join (without the cleaning) except
// it uses the path separator of c.GOOS instead of the host system's.
func (c *BuildConfig) FilePathJoin(x ...string) string {
if c.GOOS() == "windows" {
return strings.Join(x, "\\")
}
return strings.Join(x, "/")
}
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
// DistTestsExecTimeout returns how long the coordinator should wait
// for a cmd/dist test execution to run the provided dist test names.
func (c *BuildConfig) DistTestsExecTimeout(distTests []string) time.Duration {
// TODO: consider using distTests? We never did before, but
// now we have the TestStats in the coordinator. Pass in a
// *buildstats.TestStats and use historical data times some
// fudge factor? For now just use the old 20 minute limit
// we've used since 2014, but scale it by the
// GO_TEST_TIMEOUT_SCALE for the super slow builders which
// struggle with, say, the cgo tests. (which should be broken
// up into separate dist tests or shards, like the test/ dir
// was)
d := 20 * time.Minute
d *= time.Duration(c.timeoutScale())
return d
}
// timeoutScale returns this builder's GO_TEST_TIMEOUT_SCALE value, or 1.
func (c *BuildConfig) timeoutScale() int {
const pfx = "GO_TEST_TIMEOUT_SCALE="
for _, env := range [][]string{c.env, c.hostConf().env} {
for _, kv := range env {
if strings.HasPrefix(kv, pfx) {
if n, err := strconv.Atoi(kv[len(pfx):]); err == nil && n > 0 {
return n
}
}
}
}
return 1
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *BuildConfig) hostConf() *HostConfig {
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
if c.testHostConf != nil {
return c.testHostConf
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
if c, ok := Hosts[c.HostType]; ok {
return c
}
panic(fmt.Sprintf("missing buildlet config for buildlet %q", c.Name))
}
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
// BuildletBinaryURL returns the public URL of this builder's buildlet.
func (c *BuildConfig) GoBootstrapURL(e *buildenv.Environment) string {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
return strings.Replace(c.hostConf().goBootstrapURLTmpl, "$BUCKET", e.BuildletBucket, 1)
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
}
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
// BuildletBinaryURL returns the public URL of this builder's buildlet.
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *HostConfig) BuildletBinaryURL(e *buildenv.Environment) string {
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
tmpl := c.buildletURLTmpl
all: consolidate configuration for coordinator and gomote buildenv.Environment type defines configuration options: - Coordinator uses the GCE project name to lookup config. A custom config name can be provided at runtime to override. - The conventional prod and stage project names ('symbolic-datum-552' and 'go-dashboard-dev') map to prod and staging configuration structs. - Production and staging status is explicitly defined in configuration. - GCS bucket names for buildlet, logs, and snapshots are configurable. Change-Id: I7e6d7874eb0bdfe35dbdd5fcf6212ab50d576b88 Reviewed-on: https://go-review.googlesource.com/19502 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-15 02:59:59 +03:00
return strings.Replace(tmpl, "$BUCKET", e.BuildletBucket, 1)
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
}
cmd/coordinator: shard linux race builder Fixes golang/go#11074 Change-Id: I9401122e4d7febd3e38a80be89fcb2e03969f383 Reviewed-on: https://go-review.googlesource.com/11168 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-17 18:22:14 +03:00
func (c *BuildConfig) IsRace() bool {
return strings.HasSuffix(c.Name, "-race")
}
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
func (c *BuildConfig) GoInstallRacePackages() []string {
if c.InstallRacePackages != nil {
return append([]string(nil), c.InstallRacePackages...)
}
if c.IsRace() {
return []string{"std"}
}
return nil
}
dashboard: move buildlet exec code from coordinator to client package Change-Id: I778ac78ed02be9f67436ec045a3816dfc24afda3 Reviewed-on: https://go-review.googlesource.com/2923 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 20:54:03 +03:00
// AllScript returns the relative path to the operating system's script to
// do the build and run its standard set of tests.
// Example values are "src/all.bash", "src/all.bat", "src/all.rc".
func (c *BuildConfig) AllScript() string {
dashboard: add sanity check Because almost all the BuildConfig methods indirectly call AllScript, add in a little defense against accident creation of BuildConfig zero values (of the value type, not nil pointers). This can happen by doing a lookup in a map[string]dashboard.BuildConfig and then calling a method on it. We should probably move everything to pointer values of BuildConfig. There aren't many value types remaining. This paranoia isn't a result of any bug yet, but rather than the result of discussion on CL 40477. Change-Id: Ic273d4de28ac8930b56bc232bf77a9ba39f99fad Reviewed-on: https://go-review.googlesource.com/40497 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 00:37:13 +03:00
if c.Name == "" {
panic("bogus BuildConfig")
}
cmd/coordinator: shard linux race builder Fixes golang/go#11074 Change-Id: I9401122e4d7febd3e38a80be89fcb2e03969f383 Reviewed-on: https://go-review.googlesource.com/11168 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-17 18:22:14 +03:00
if c.IsRace() {
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
if strings.HasPrefix(c.Name, "windows-") {
return "src/race.bat"
}
return "src/race.bash"
}
dashboard: move buildlet exec code from coordinator to client package Change-Id: I778ac78ed02be9f67436ec045a3816dfc24afda3 Reviewed-on: https://go-review.googlesource.com/2923 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 20:54:03 +03:00
if strings.HasPrefix(c.Name, "windows-") {
return "src/all.bat"
}
if strings.HasPrefix(c.Name, "plan9-") {
return "src/all.rc"
}
env: update nacl builders to be VM-based Be lazy and use sid as a base, instead of trusty. This replaces my earlier failed attempt: https://golang.org/cl/4960 Fixes golang/go#9879 Change-Id: Ia0a39c99747844b6e34bf7561c01dce49fa3b226 Reviewed-on: https://go-review.googlesource.com/5191 Reviewed-by: Russ Cox <rsc@golang.org>
2015-02-19 01:12:22 +03:00
if strings.HasPrefix(c.Name, "nacl-") {
return "src/nacltest.bash"
}
dashboard: add darwin-arm/arm64 buildlets Tested by running a local coordinator on the darwin-arm64 builder. Change-Id: Id38faaa3275d6535b897f3c65df6d01ed9c1847e Reviewed-on: https://go-review.googlesource.com/9459 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-04-29 15:54:19 +03:00
if strings.HasPrefix(c.Name, "darwin-arm") {
return "src/iostest.bash"
}
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
if strings.HasPrefix(c.Name, "misc-compile") {
dashboard: add all-compile builder Runs buildall.bash only. Change-Id: I1a68d54569ccb511d956c5b181e463503353d4d4 Reviewed-on: https://go-review.googlesource.com/9512 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2015-04-30 00:44:46 +03:00
return "src/buildall.bash"
}
dashboard: move buildlet exec code from coordinator to client package Change-Id: I778ac78ed02be9f67436ec045a3816dfc24afda3 Reviewed-on: https://go-review.googlesource.com/2923 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 20:54:03 +03:00
return "src/all.bash"
}
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
// SplitMakeRun reports whether the coordinator should first compile
// (using c.MakeScript), then snapshot, then run the tests (ideally
// sharded) using c.RunScript.
// Eventually this function should always return true (and then be deleted)
// but for now we've only set up the scripts and verified that the main
// configurations work.
func (c *BuildConfig) SplitMakeRun() bool {
switch c.AllScript() {
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
case "src/all.bash", "src/all.bat",
"src/race.bash", "src/race.bat",
"src/all.rc",
"src/nacltest.bash":
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
// These we've verified to work.
return true
}
dashboard: remove eliasnaur Android emulators Now that we hace Android emulator builders on GCE, drop the old ones running on hardware. Also update the configuration of the remaining builders now that androidtest.bash is no longer necessary. Change-Id: I09cab08b28952f967205d6c35f36eb6e05a17950 Reviewed-on: https://go-review.googlesource.com/c/164397 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 10:19:36 +03:00
// TODO(bradfitz): make iostest.bash work too. And
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
// buildall.bash should really just be N small container jobs
// instead of a "buildall.bash". Then we can delete this whole
// method.
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
return false
}
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
func (c *BuildConfig) buildSubrepos() bool {
cmd/coordinator: build sub-repositories Fixes golang/go#9506 Change-Id: Iebfb52386b5ba3df7d058f266e227d634d210ec6 Reviewed-on: https://go-review.googlesource.com/10732 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-05 02:45:17 +03:00
if !c.SplitMakeRun() {
return false
}
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
// TODO(bradfitz,dmitshur): move this into BuildConfig bools, rather than this Name switch.
dashboard: build sub-repos on more builders Change-Id: I57c12f427448281d71b8507ed950038209134143 Reviewed-on: https://go-review.googlesource.com/10915 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-11 20:01:24 +03:00
switch c.Name {
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
case "darwin-amd64-10_11",
cmd/coordinator, dashboard: switch to new VMWare-based OS X builders Ignore the old darwin-{amd64,386}-10_10 builders. Don't give them an error, but pretend they don't exist. Also: switch trybots from OS X 10.10 to OS X 10.11, and re-enable sharding. Let's hope for the best. See golang/go#12979. This also enables subrepo tests for all OS X versions. darwin-386-* is currently offline, pending some golang/go#17009 Updates golang/go#9495 (OS X virtualization) Change-Id: I4d53a79087404b5e8051d1aff0c668a92625f442 Reviewed-on: https://go-review.googlesource.com/28583 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-07 08:41:02 +03:00
"darwin-386-10_11",
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
// TODO: add darwin-amd64-10_12 when we have a build scheduler
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
"freebsd-amd64-93",
"freebsd-386-10_3", "freebsd-amd64-10_3",
"freebsd-386-11_1", "freebsd-amd64-11_1",
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
"freebsd-386-10_4", "freebsd-amd64-10_4",
"freebsd-386-11_2", "freebsd-amd64-11_2",
dashboard: build sub-repos on more builders Change-Id: I57c12f427448281d71b8507ed950038209134143 Reviewed-on: https://go-review.googlesource.com/10915 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-11 20:01:24 +03:00
"linux-386", "linux-amd64", "linux-amd64-nocgo",
dashboard: test sub-repositories on s390x Fixes golang/go#26131 Change-Id: I07e84c255a158b843168612cc3dad7688343ffb7 Reviewed-on: https://go-review.googlesource.com/127375 Reviewed-by: Michael Munday <mike.munday@ibm.com> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-08-01 23:08:56 +03:00
"linux-s390x-ibm",
dashboard: switch to OpenBSD 6.0 completely, remove OpenBSD 5.8 Updates golang/go#16951 Change-Id: I7b92363671e0ba4c6ab80cef60e7c8600687f440 Reviewed-on: https://go-review.googlesource.com/34384 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-12-15 01:41:25 +03:00
"openbsd-386-60", "openbsd-amd64-60",
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
"openbsd-386-62", "openbsd-amd64-62",
dashboard: add OpenBSD 6.4 to BuildConfig.buildSubrepos By now, OpenBSD 6.4 has proven reliable, so we want it to be used for building subrepos as well. Updates golang/go#26833 Change-Id: I026207865514a026f86936140a16dd707f9594ea Reviewed-on: https://go-review.googlesource.com/c/148888 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-10 21:59:23 +03:00
"openbsd-386-64", "openbsd-amd64-64",
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
"netbsd-amd64-8_0",
"netbsd-386-8_0",
dashboard: build sub-repos on more builders Change-Id: I57c12f427448281d71b8507ed950038209134143 Reviewed-on: https://go-review.googlesource.com/10915 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-11 20:01:24 +03:00
"plan9-386",
dashboard: test subrepos for freebsd-arm, add more misc-compile builders Change-Id: I51f3fee3408d99a502887ea6802f6979feb36439 Reviewed-on: https://go-review.googlesource.com/54051 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-09 03:53:04 +03:00
"freebsd-arm-paulzhol",
dashboard: update Windows builders, remove old manual Windows builder And clean up some other builders. Change-Id: I1f71836da7008e58d3e76e2cc3170e96cd57ddf6 Reviewed-on: https://go-review.googlesource.com/51432 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-27 07:52:12 +03:00
"windows-amd64-2016", "windows-386-2008":
dashboard: build sub-repos on more builders Change-Id: I57c12f427448281d71b8507ed950038209134143 Reviewed-on: https://go-review.googlesource.com/10915 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-11 20:01:24 +03:00
return true
default:
return false
}
cmd/coordinator: build sub-repositories Fixes golang/go#9506 Change-Id: Iebfb52386b5ba3df7d058f266e227d634d210ec6 Reviewed-on: https://go-review.googlesource.com/10732 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-05 02:45:17 +03:00
}
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
// BuildRepo reports whether we should do post-submit builds of the provided
// repo ("go", "sys", "net", etc).
func (c *BuildConfig) BuildRepo(repo string) bool {
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
if p := c.PostSubmitBuildPolicy; p != nil {
return p(repo)
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
}
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
if repo == "go" {
dashboard: fix android/mobile trybot policy, add tests To fix what happened in https://go-review.googlesource.com/c/mobile/+/164177 where it ran linux-amd64 trybots against the past two Go releases (Go 1.11 and Go 1.12), despite that builder not being able to run mobile tests. Change-Id: Ib5150a0dc6b30cf855884a97b29ab2c79b423096 Reviewed-on: https://go-review.googlesource.com/c/164682 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 05:12:29 +03:00
return true
}
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
return c.buildSubrepos()
}
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
// BuildBranch reports whether we should do post-submit builds of the provided
// branch.
// repo is "go", "sys", "net", etc.
// branch is the branch of the repo (usually "master").
// goBranch is non-empty for a non-"go" repo, and is the branch of Go the subrepo is being tested at.
func (c *BuildConfig) BuildBranch(repo, branch, goBranch string) bool {
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
// Don't build old branches.
const minGo1x = 11
if strings.HasPrefix(goBranch, "release-branch.go1") {
if v, _ := strconv.Atoi(strings.TrimPrefix(goBranch, "release-branch.go1.")); v < minGo1x {
cmd/coordinator: support testing subrepos that use modules Only the "oauth2" and "build" repos for now, as a test. We'll lock down policy more later and decide when to do this automatically. Also, this currently only runs buildlets which run in our GCP project, because we're not yet proxying the a localhost:3000 port from the reverse buildlets to an authenticated TLS connection back to our module proxy service on GKE. Updates golang/go#14594 Fixes golang/go#29637 Change-Id: I6f05da2186b38dc8056081252563a82c50f0ce05 Reviewed-on: https://go-review.googlesource.com/c/157438 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-11 01:00:51 +03:00
return false
}
}
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
if strings.HasPrefix(branch, "release-branch.go1") {
if v, _ := strconv.Atoi(strings.TrimPrefix(branch, "release-branch.go1.")); v < minGo1x {
return false
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
}
}
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
// Skip Android builds on release branches prior to Go 1.13.
// We changed it from using androidtest.bash to the normal
// cmd/dist-based flow and we don't want to teach the build
// system exactly which revs everywhere use which path.
// Instead, just skip Android stuff on release branches.
// Android basically only lives at head anyway.
dashboard: fix android/mobile trybot policy, add tests To fix what happened in https://go-review.googlesource.com/c/mobile/+/164177 where it ran linux-amd64 trybots against the past two Go releases (Go 1.11 and Go 1.12), despite that builder not being able to run mobile tests. Change-Id: Ib5150a0dc6b30cf855884a97b29ab2c79b423096 Reviewed-on: https://go-review.googlesource.com/c/164682 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 05:12:29 +03:00
if strings.HasPrefix(c.Name, "android-") || repo == "mobile" {
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
if strings.HasPrefix(branch, "dev.") {
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
return false
}
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
b := goBranch
if b == "" {
b = branch
}
if strings.HasPrefix(b, "release-branch.go1.") {
if v, _ := strconv.Atoi(strings.TrimPrefix(b, "release-branch.go1.")); v < 13 {
return false
}
}
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
}
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
dashboard: skip dev.boringcrypto branches on most builders We care most about testing the dev.boringcrypto branches on linux/amd64. Stop testing them on all builders, as it's not worth the resource use. Continue to test on one additional non-linux/amd64 port, namely windows/386, to ensure that fallbacks work. Fixes golang/go#26791 Change-Id: I4556886f55175173de52e35cf57833b22843d586 Reviewed-on: https://go-review.googlesource.com/c/149739 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Filippo Valsorda <filippo@golang.org>
2018-11-15 09:15:27 +03:00
// Build dev.boringcrypto branches only on linux/amd64 and windows/386 (see golang.org/issue/26791).
if repo == "go" && (branch == "dev.boringcrypto" || strings.HasPrefix(branch, "dev.boringcrypto.")) {
if c.Name != "linux-amd64" && c.Name != "windows-386-2008" {
return false
}
}
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
return true
}
cmd/coordinator, dashboard: split build into 'make' and 'run' steps Change-Id: Ib60a601ef21163d5e0d1fed3a2609d9b719c616a Reviewed-on: https://go-review.googlesource.com/10073 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-21 07:02:38 +03:00
// AllScriptArgs returns the set of arguments that should be passed to the
dashboard: add darwin-arm/arm64 buildlets Tested by running a local coordinator on the darwin-arm64 builder. Change-Id: Id38faaa3275d6535b897f3c65df6d01ed9c1847e Reviewed-on: https://go-review.googlesource.com/9459 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-04-29 15:54:19 +03:00
// all.bash-equivalent script. Usually empty.
func (c *BuildConfig) AllScriptArgs() []string {
if strings.HasPrefix(c.Name, "darwin-arm") {
return []string{"-restart"}
}
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
return append([]string(nil), c.allScriptArgs...)
dashboard: add darwin-arm/arm64 buildlets Tested by running a local coordinator on the darwin-arm64 builder. Change-Id: Id38faaa3275d6535b897f3c65df6d01ed9c1847e Reviewed-on: https://go-review.googlesource.com/9459 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-04-29 15:54:19 +03:00
}
cmd/release: add new program to build releases The release command will replace misc/makerelease from the main repo. The contents of makerelease make their way into this program. Also add GorootFinal and MakeScript methods to dashboard.BuildConfig. Change-Id: Id20f3f7a85a21f040d8cf47e438d299d57360556 Reviewed-on: https://go-review.googlesource.com/3693 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-02 15:05:01 +03:00
// MakeScript returns the relative path to the operating system's script to
// do the build.
// Example values are "src/make.bash", "src/make.bat", "src/make.rc".
func (c *BuildConfig) MakeScript() string {
if strings.HasPrefix(c.Name, "windows-") {
return "src/make.bat"
}
if strings.HasPrefix(c.Name, "plan9-") {
return "src/make.rc"
}
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
if strings.HasPrefix(c.Name, "nacl-") {
return "src/naclmake.bash"
}
cmd/release: add new program to build releases The release command will replace misc/makerelease from the main repo. The contents of makerelease make their way into this program. Also add GorootFinal and MakeScript methods to dashboard.BuildConfig. Change-Id: Id20f3f7a85a21f040d8cf47e438d299d57360556 Reviewed-on: https://go-review.googlesource.com/3693 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-02 15:05:01 +03:00
return "src/make.bash"
}
cmd/coordinator, dashboard: split build into 'make' and 'run' steps Change-Id: Ib60a601ef21163d5e0d1fed3a2609d9b719c616a Reviewed-on: https://go-review.googlesource.com/10073 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-21 07:02:38 +03:00
// MakeScriptArgs returns the set of arguments that should be passed to the
// make.bash-equivalent script. Usually empty.
func (c *BuildConfig) MakeScriptArgs() []string {
return c.AllScriptArgs()
}
// RunScript returns the relative path to the operating system's script to
// run the test suite.
// Example values are "src/run.bash", "src/run.bat", "src/run.rc".
func (c *BuildConfig) RunScript() string {
if strings.HasPrefix(c.Name, "windows-") {
return "src/run.bat"
}
if strings.HasPrefix(c.Name, "plan9-") {
return "src/run.rc"
}
return "src/run.bash"
}
// RunScriptArgs returns the set of arguments that should be passed to the
// run.bash-equivalent script.
func (c *BuildConfig) RunScriptArgs() []string {
return []string{"--no-rebuild"}
}
cmd/release: add new program to build releases The release command will replace misc/makerelease from the main repo. The contents of makerelease make their way into this program. Also add GorootFinal and MakeScript methods to dashboard.BuildConfig. Change-Id: Id20f3f7a85a21f040d8cf47e438d299d57360556 Reviewed-on: https://go-review.googlesource.com/3693 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-02 15:05:01 +03:00
// GorootFinal returns the default install location for
// releases for this platform.
func (c *BuildConfig) GorootFinal() string {
if strings.HasPrefix(c.Name, "windows-") {
return "c:\\go"
}
return "/usr/local/go"
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
// MachineType returns the GCE machine type to use for this builder.
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *HostConfig) MachineType() string {
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
if v := c.machineType; v != "" {
return v
}
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
if c.IsContainer() {
// Set a higher default machine size for containers,
// so their /workdir tmpfs can be larger. The COS
// image has no swap, so we want to make sure the
// /workdir fits completely in memory.
return "n1-standard-4" // 4 CPUs, 15GB RAM
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
return "n1-highcpu-2"
}
cmd/coordinator: generate table of builders To replace the always-out-of-date wiki. Change-Id: I63d56d4bcf9252e56c833eaa15656b0321ac91dc Reviewed-on: https://go-review.googlesource.com/9571 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-01 03:03:01 +03:00
// ShortOwner returns a short human-readable owner.
func (c BuildConfig) ShortOwner() string {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
owner := c.hostConf().Owner
if owner == "" {
cmd/coordinator: generate table of builders To replace the always-out-of-date wiki. Change-Id: I63d56d4bcf9252e56c833eaa15656b0321ac91dc Reviewed-on: https://go-review.googlesource.com/9571 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-01 03:03:01 +03:00
return "go-dev"
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
return strings.TrimSuffix(owner, "@golang.org")
cmd/coordinator: generate table of builders To replace the always-out-of-date wiki. Change-Id: I63d56d4bcf9252e56c833eaa15656b0321ac91dc Reviewed-on: https://go-review.googlesource.com/9571 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-01 03:03:01 +03:00
}
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
// OwnerGithub returns the Github handle of the owner.
func (c BuildConfig) OwnerGithub() string {
return c.hostConf().OwnerGithub
}
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
// PoolName returns a short summary of the builder's host type for the
all: upgrade farmer.golang.org links to https Since farmer.golang.org has a valid certificate in production now, we should use that certificate for retrieving data. Change-Id: I3685d84953ea11e6ac09b4692fa27c73538a565c Reviewed-on: https://go-review.googlesource.com/39290 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-03 17:51:28 +03:00
// https://farmer.golang.org/builders page.
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
func (c *HostConfig) PoolName() string {
switch {
case c.IsReverse:
return "Reverse (dedicated machine/VM)"
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsVM():
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
return "GCE VM"
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsContainer():
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
return "Container"
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
}
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
panic("unknown builder type")
}
env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx This adds scripts to create a new builder host image that acts like Container-Optimized OS (has docker, runs konlet on startup) but with a Debian 9 kernel + userspace that permits KVM for nested virtualization. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884 Reviewed-on: https://go-review.googlesource.com/c/163301 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-22 00:10:54 +03:00
// ContainerVMImage returns the base VM name (not the fully qualified
// URL resource name of the VM) that starts the konlet program that
// pulls & runs a container. This method is only applicable when
// c.IsContainer() is true.
func (c *HostConfig) ContainerVMImage() string {
if c.KonletVMImage != "" {
return c.KonletVMImage
}
return "debian-stretch-vmx"
}
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
// IsHermetic reports whether this host config gets a fresh
// environment (including /usr, /var, etc) for each execution. This is
// true for VMs, GKE, and reverse buildlets running their containers
// running in Docker, but false on some reverse buildlets.
func (c *HostConfig) IsHermetic() bool {
switch {
case c.IsReverse:
return c.HermeticReverse
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsVM():
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
return true
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsContainer():
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
return true
}
panic("unknown builder type")
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
}
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
// GCENumCPU reports the number of GCE CPUs this buildlet requires.
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *HostConfig) GCENumCPU() int {
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
t := c.MachineType()
n, _ := strconv.Atoi(t[strings.LastIndex(t, "-")+1:])
return n
}
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
func (c *BuildConfig) NumTestHelpers(isTry bool) int {
if isTry && c.numTryTestHelpers != 0 {
return c.numTryTestHelpers
}
return c.numTestHelpers
}
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
// defaultTrySet returns a trybot policy function that reports whether
// a project should use trybots. All the default projects are included,
// plus any given in extraProj.
func defaultTrySet(extraProj ...string) func(proj string) bool {
return func(proj string) bool {
if proj == "go" {
return true
}
for _, p := range extraProj {
if proj == p {
return true
}
}
cmd/coordinator: support testing subrepos that use modules Only the "oauth2" and "build" repos for now, as a test. We'll lock down policy more later and decide when to do this automatically. Also, this currently only runs buildlets which run in our GCP project, because we're not yet proxying the a localhost:3000 port from the reverse buildlets to an authenticated TLS connection back to our module proxy service on GKE. Updates golang/go#14594 Fixes golang/go#29637 Change-Id: I6f05da2186b38dc8056081252563a82c50f0ce05 Reviewed-on: https://go-review.googlesource.com/c/157438 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-11 01:00:51 +03:00
// TODO: remove items from this set once these repos have go.mod files:
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
switch proj {
cmd/coordinator: support testing subrepos that use modules Only the "oauth2" and "build" repos for now, as a test. We'll lock down policy more later and decide when to do this automatically. Also, this currently only runs buildlets which run in our GCP project, because we're not yet proxying the a localhost:3000 port from the reverse buildlets to an authenticated TLS connection back to our module proxy service on GKE. Updates golang/go#14594 Fixes golang/go#29637 Change-Id: I6f05da2186b38dc8056081252563a82c50f0ce05 Reviewed-on: https://go-review.googlesource.com/c/157438 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-11 01:00:51 +03:00
case "grpc-review", "exp", "mobile", "term":
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
return false
}
return true
}
}
// explicitTrySet returns a trybot policy function that reports
// whether a project should use trybots. Only the provided projects in
// projs are enabled.
func explicitTrySet(projs ...string) func(proj string) bool {
return func(proj string) bool {
for _, p := range projs {
if proj == p {
return true
}
}
return false
}
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
func init() {
dashboard, env: add FreeBSD 9.3 support. Differences from 10.1 support: - Uses gcc in base image, 10.1 uses clang. - Uses dig in startup diagnostic, 10.1 uses drill. Fixes #8639 Change-Id: I298bdf7d76c91395ae56606d1e2f0176eb0eb792 Reviewed-on: https://go-review.googlesource.com/3173 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-22 08:16:54 +03:00
addBuilder(BuildConfig{
dashboard: reenable FreeBSD 9.3/10.1, OpenBSD, and Plan9 builders The GCE bug is fixed and these VMs boot again. (Previously: golang.org/cl/50011) Change-Id: I3f330b8d0c6a6ae6708b2299bba13945a06fa36f Reviewed-on: https://go-review.googlesource.com/53270 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-04 07:25:07 +03:00
Name: "freebsd-amd64-gce93",
HostType: "host-freebsd-93-gce",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
TryOnly: true, // don't run regular build...
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
dashboard, env: add FreeBSD 9.3 support. Differences from 10.1 support: - Uses gcc in base image, 10.1 uses clang. - Uses dig in startup diagnostic, 10.1 uses drill. Fixes #8639 Change-Id: I298bdf7d76c91395ae56606d1e2f0176eb0eb792 Reviewed-on: https://go-review.googlesource.com/3173 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-22 08:16:54 +03:00
})
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
addBuilder(BuildConfig{
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
Name: "freebsd-amd64-10_3",
HostType: "host-freebsd-10_3",
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
})
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
addBuilder(BuildConfig{
Name: "freebsd-amd64-10_4",
HostType: "host-freebsd-10_4",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("sys"),
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
MaxAtOnce: 2,
})
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
addBuilder(BuildConfig{
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
Name: "freebsd-amd64-11_1",
HostType: "host-freebsd-11_1",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil,
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
ShouldRunDistTest: fasterTrybots,
dashboard: update Windows builders, remove old manual Windows builder And clean up some other builders. Change-Id: I1f71836da7008e58d3e76e2cc3170e96cd57ddf6 Reviewed-on: https://go-review.googlesource.com/51432 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-27 07:52:12 +03:00
numTryTestHelpers: 4,
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
})
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
addBuilder(BuildConfig{
Name: "freebsd-amd64-11_2",
HostType: "host-freebsd-11_2",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("sys"),
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
ShouldRunDistTest: fasterTrybots,
numTryTestHelpers: 4,
MaxAtOnce: 2,
})
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
addBuilder(BuildConfig{
Name: "freebsd-amd64-12_0",
HostType: "host-freebsd-12_0",
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
MinimumGoVersion: types.MajorMinor{1, 11},
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
ShouldRunDistTest: fasterTrybots,
numTryTestHelpers: 4,
MaxAtOnce: 2,
})
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
addBuilder(BuildConfig{
dashboard: reenable FreeBSD 9.3/10.1, OpenBSD, and Plan9 builders The GCE bug is fixed and these VMs boot again. (Previously: golang.org/cl/50011) Change-Id: I3f330b8d0c6a6ae6708b2299bba13945a06fa36f Reviewed-on: https://go-review.googlesource.com/53270 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-04 07:25:07 +03:00
Name: "freebsd-amd64-race",
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
HostType: "host-freebsd-11_1",
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
})
addBuilder(BuildConfig{
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
Name: "freebsd-386-10_3",
HostType: "host-freebsd-10_3",
dashboard: reenable FreeBSD 9.3/10.1, OpenBSD, and Plan9 builders The GCE bug is fixed and these VMs boot again. (Previously: golang.org/cl/50011) Change-Id: I3f330b8d0c6a6ae6708b2299bba13945a06fa36f Reviewed-on: https://go-review.googlesource.com/53270 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-04 07:25:07 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
addBuilder(BuildConfig{
Name: "freebsd-386-10_4",
HostType: "host-freebsd-10_4",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("sys"),
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
MaxAtOnce: 2,
})
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "freebsd-386-11_1",
HostType: "host-freebsd-11_1",
ShouldRunDistTest: noTestDir,
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
MaxAtOnce: 2,
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
})
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
addBuilder(BuildConfig{
Name: "freebsd-386-11_2",
HostType: "host-freebsd-11_2",
ShouldRunDistTest: noTestDir,
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
MaxAtOnce: 2,
})
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "linux-386",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
ShouldRunDistTest: fasterTrybots,
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
numTestHelpers: 1,
numTryTestHelpers: 3,
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "linux-386-387",
Notes: "GO386=387",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386", "GO386=387"},
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-amd64",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 3,
numTestHelpers: 1,
numTryTestHelpers: 4,
RunBench: true,
all: run dev coordinator locally, allow debug in prod * flag allows debug (trigger a build) in prod mode * use default zone (us-central1-f) in staging * compute.CloudPlatformScope satisfies compute and storage scope requirements * new buildlet for linux-amd64 on Kubernetes Updates golang/go#12546 Change-Id: I1a0cbc0e0c0552b7ac56943c646378194508d48f Reviewed-on: https://go-review.googlesource.com/17102 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-11-20 21:00:06 +03:00
})
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-vmx",
HostType: "host-linux-stretch-vmx",
MaxAtOnce: 1,
TryOnly: true, // don't run regular build
tryBot: nil, // and don't run trybots (only gomote)
})
dashboard: remove alpine builder It's doing no good being red all the time, especially as nobody is working on it. Updates golang/go#22689 Updates golang/go#19938 Change-Id: Icb947878d85e920f24ea458eb0f319844ca5bd60 Reviewed-on: https://go-review.googlesource.com/78575 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-18 06:31:35 +03:00
const testAlpine = false // Issue 22689 (hide all red builders), Issue 19938 (get Alpine passing)
if testAlpine {
addBuilder(BuildConfig{
Name: "linux-amd64-alpine",
HostType: "host-linux-x86-alpine",
})
}
dashboard, coordinator: add vetall builder Change-Id: I331a27050e6f068aa47afa9012a9bb50e3b0c46a Reviewed-on: https://go-review.googlesource.com/37573 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-01-09 23:02:06 +03:00
// Add the -vetall builder. The builder name suffix "-vetall" is recognized by cmd/dist/test.go
// to only run the "go vet std cmd" test and no others.
addBuilder(BuildConfig{
Name: "misc-vet-vetall",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
dashboard, coordinator: add vetall builder Change-Id: I331a27050e6f068aa47afa9012a9bb50e3b0c46a Reviewed-on: https://go-review.googlesource.com/37573 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-01-09 23:02:06 +03:00
Notes: "Runs vet over the standard library.",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
dashboard, coordinator: add vetall builder Change-Id: I331a27050e6f068aa47afa9012a9bb50e3b0c46a Reviewed-on: https://go-review.googlesource.com/37573 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-01-09 23:02:06 +03:00
numTestHelpers: 5,
})
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
addMiscCompile := func(suffix, rx string) {
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "misc-compile" + suffix,
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
TryOnly: true,
CompileOnly: true,
Notes: "Runs buildall.sh to cross-compile std packages for " + rx + ", but doesn't run any tests.",
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
allScriptArgs: []string{
// Filtering pattern to buildall.bash:
rx,
},
})
}
dashboard: add more misc-compile trybots s390x notably was missing. Also add the mobile ones. Change-Id: Id6be52b73fac897980d4e94ac5c5b07027f5b4e7 Reviewed-on: https://go-review.googlesource.com/114315 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-24 01:19:10 +03:00
addMiscCompile("", "^(linux-arm64|linux-s390x|solaris-amd64|darwin-386)$") // 4 ports
addMiscCompile("-nacl", "^nacl") // 3
addMiscCompile("-mips", "^linux-mips") // 4
addMiscCompile("-ppc", "^linux-ppc64") // 2
addMiscCompile("-plan9", "^plan9-") // 3
addMiscCompile("-freebsd", "^freebsd-") // 3
addMiscCompile("-netbsd", "^netbsd-") // 3
addMiscCompile("-openbsd", "^openbsd-") // 3
dashboard: remove mobile misc-compile trybots I guess they never worked. Updates golang/go#25963 Change-Id: I1409a704670032410c3c6b87cbbb876fc96dfb22 Reviewed-on: https://go-review.googlesource.com/119856 Reviewed-by: Filippo Valsorda <filippo@golang.org>
2018-06-19 23:50:06 +03:00
// TODO: Issue 25963, get the misc-compile trybots for mobile working, and then:
// addMiscCompile("-mobile", "(^android|darwin-arm64)") // 5 ports
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-amd64-nocgo",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 1,
Notes: "cgo disabled",
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
env: []string{
"CGO_ENABLED=0",
// This USER=root was required for Docker-based builds but probably isn't required
// in the VM anymore, since the buildlet probably already has this in its environment.
// (It was required because without cgo, it couldn't find the username)
"USER=root",
},
})
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-amd64-noopt",
Notes: "optimizations and inlining disabled",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
env: []string{"GO_GCFLAGS=-N -l"},
MaxAtOnce: 1,
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
dashboard: add SSA check builder Now that SSA checks are off by default, let's run a builder with the checks on just to make sure they keep passing. Change-Id: I7edd2a2865681a135ccf755be7a14f8645c0a8aa Reviewed-on: https://go-review.googlesource.com/20882 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-19 01:30:38 +03:00
addBuilder(BuildConfig{
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
Name: "linux-amd64-ssacheck",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 1,
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil, // TODO: add a func to conditionally run this trybot if compiler dirs are touched
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
CompileOnly: true,
Notes: "SSA internal checks enabled",
dashboard: add -d=dclstack to ssacheck builder Fixes golang/go#18621 Change-Id: I362ab473136cce659fa267d5bfc0a670389e3fc4 Reviewed-on: https://go-review.googlesource.com/38321 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-17 07:55:29 +03:00
env: []string{"GO_GCFLAGS=-d=ssa/check/on,dclstack"},
cmd/coordinator: move git ancestor checking before buildlet creation, generalize This also adds GoDeps to the builders config file and adds a dependency for the ssacheck builder, which currently fails on older release branches with: > Failed on linux-amd64-ssacheck: > https://storage.googleapis.com/go-build-log/1255edaf/linux-amd64-ssacheck_18f7e015.log > > ... > compile: unknown debug key -d dclstack Change-Id: I4ad0a298705feed56f467f3c8d9488f8be2e4af5 Reviewed-on: https://go-review.googlesource.com/43774 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-20 00:23:14 +03:00
GoDeps: []string{
"f65abf6ddc8d1f3d403a9195fd74eaffa022b07f", // adds dclstack
},
dashboard: add SSA check builder Now that SSA checks are off by default, let's run a builder with the checks on just to make sure they keep passing. Change-Id: I7edd2a2865681a135ccf755be7a14f8645c0a8aa Reviewed-on: https://go-review.googlesource.com/20882 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-19 01:30:38 +03:00
})
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-racecompile",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil, // TODO: add a func to conditionally run this trybot if compiler dirs are touched
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 1,
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
CompileOnly: true,
cmd/coordinator: make linux-amd64-racecompile rebuild std/cmd with -c=128 Updates golang/go#19962 Change-Id: I8b938a39b33d4e6b9b8dc5bc0802da95e889ee46 Reviewed-on: https://go-review.googlesource.com/41613 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-25 00:14:05 +03:00
SkipSnapshot: true,
StopAfterMake: true,
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
InstallRacePackages: []string{"cmd/compile"},
Notes: "race-enabled cmd/compile",
cmd/coordinator: move git ancestor checking before buildlet creation, generalize This also adds GoDeps to the builders config file and adds a dependency for the ssacheck builder, which currently fails on older release branches with: > Failed on linux-amd64-ssacheck: > https://storage.googleapis.com/go-build-log/1255edaf/linux-amd64-ssacheck_18f7e015.log > > ... > compile: unknown debug key -d dclstack Change-Id: I4ad0a298705feed56f467f3c8d9488f8be2e4af5 Reviewed-on: https://go-review.googlesource.com/43774 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-20 00:23:14 +03:00
GoDeps: []string{
"22f1b56dab29d397d2bdbdd603d85e60fb678089", // adds cmd/compile -c; Issue 20222
},
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
})
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
Name: "linux-amd64-race",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 1,
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
ShouldRunDistTest: fasterTrybots,
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
numTestHelpers: 1,
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
numTryTestHelpers: 5,
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
})
env: convert linux-x86-clang to be VM-based Change-Id: I583d751652dec9176169b4d4705f3813506017e3 Reviewed-on: https://go-review.googlesource.com/4790 Reviewed-by: Minux Ma <minux@golang.org>
2015-02-13 06:45:02 +03:00
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-386-clang",
HostType: "host-linux-clang",
MaxAtOnce: 1,
Notes: "Debian jessie + clang 3.9 instead of gcc",
env: []string{"CC=/usr/bin/clang", "GOHOSTARCH=386"},
env: convert linux-x86-clang to be VM-based Change-Id: I583d751652dec9176169b4d4705f3813506017e3 Reviewed-on: https://go-review.googlesource.com/4790 Reviewed-by: Minux Ma <minux@golang.org>
2015-02-13 06:45:02 +03:00
})
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-amd64-clang",
HostType: "host-linux-clang",
MaxAtOnce: 1,
Notes: "Debian jessie + clang 3.9 instead of gcc",
env: []string{"CC=/usr/bin/clang"},
env: convert linux-x86-clang to be VM-based Change-Id: I583d751652dec9176169b4d4705f3813506017e3 Reviewed-on: https://go-review.googlesource.com/4790 Reviewed-by: Minux Ma <minux@golang.org>
2015-02-13 06:45:02 +03:00
})
env: update linux-x86-sid to be a VM Fixes #9873 Change-Id: I08c501ef52ebde4cdbc48fe45a5342488000c433 Reviewed-on: https://go-review.googlesource.com/4874 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-14 06:01:32 +03:00
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-386-sid",
HostType: "host-linux-sid",
Notes: "Debian sid (unstable)",
MaxAtOnce: 1,
env: []string{"GOHOSTARCH=386"},
env: update linux-x86-sid to be a VM Fixes #9873 Change-Id: I08c501ef52ebde4cdbc48fe45a5342488000c433 Reviewed-on: https://go-review.googlesource.com/4874 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-14 06:01:32 +03:00
})
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "linux-amd64-sid",
HostType: "host-linux-sid",
MaxAtOnce: 1,
Notes: "Debian sid (unstable)",
env: update linux-x86-sid to be a VM Fixes #9873 Change-Id: I08c501ef52ebde4cdbc48fe45a5342488000c433 Reviewed-on: https://go-review.googlesource.com/4874 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-14 06:01:32 +03:00
})
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-androidemu",
HostType: "host-android-amd64-emu",
env: []string{
"GOARCH=amd64",
"GOOS=linux",
"CGO_ENABLED=1",
},
tryBot: func(proj string) bool {
// Only for mobile repo for now, not "go":
return proj == "mobile"
},
PostSubmitBuildPolicy: func(repo string) bool {
return repo == "mobile"
},
Notes: "Runs GOOS=linux but with the Android emulator attached, for running x/mobile host tests.",
})
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-stretch",
HostType: "host-linux-stretch",
MaxAtOnce: 1,
Notes: "Debian Stretch",
})
dashboard: add "longtest" builder to run all tests, without -short mode Fixes golang/go#12508 Change-Id: I62e1e296f10c7622c1967a5530c4b0e07c136dfd Reviewed-on: https://go-review.googlesource.com/113436 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-16 22:50:39 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-longtest",
HostType: "host-linux-stretch",
MaxAtOnce: 1,
Notes: "Debian Stretch with go test -short=false",
dashboard: increase TIMEOUT_SCALE to 2 for linux-amd64-longtest runtime tests take about 180s in long mode, too close to the timeout limit (180 seconds), and this is causing sporadic timeout failures like https://build.golang.org/log/cda568a39f396e3fee64778a475528be452632ed Bump up the builder's timeout by 2x. Fixes golang/go#25629 Change-Id: I77e0f86f0b236cb953457ebdcbfc5763c2c6d54b Reviewed-on: https://go-review.googlesource.com/115016 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-29 22:03:23 +03:00
env: []string{
"GO_TEST_SHORT=0",
// runtime takes ~190 seconds in long mode, which is over
// the 180 seconds default timeout.
"GO_TEST_TIMEOUT_SCALE=2",
},
dashboard: add "longtest" builder to run all tests, without -short mode Fixes golang/go#12508 Change-Id: I62e1e296f10c7622c1967a5530c4b0e07c136dfd Reviewed-on: https://go-review.googlesource.com/113436 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-16 22:50:39 +03:00
})
dashboard: add linux-arm-qemu builder (disabled) and adjust plan9 builder Change-Id: If2b4074f9bc1b44e97897096eb173b42c8323250 Reviewed-on: https://go-review.googlesource.com/6411 Reviewed-by: Salman Aljammaz <salman.aljammaz@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-03-01 20:23:57 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
Name: "linux-arm",
all: dockerize scaleway builders Fixes golang/go#20397 Change-Id: I353e879e61aeef56f0c228bbd37cc1eea2ce8b4e Reviewed-on: https://go-review.googlesource.com/43612 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-18 00:28:46 +03:00
HostType: "host-linux-arm-scaleway",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil, // Issue 22748, Issue 22749
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
FlakyNet: true,
numTestHelpers: 2,
numTryTestHelpers: 7,
dashboard, cmd/coordinator: sharded ARM trybot-only builders This is a quick & hacky version of our eventual sharding plan. Later we want to run make.bash once, snapshot it, and then intelligently schedule all the tests out over N machines (starting with the built snapshot), giving each the next test to run as they become idle. And then we want to stitch together all of their output. But this CL just makes 6 ARM builders, each of which runs make.bash (4 minutes) and then each which runs a statically-configured subset of the tests. Based on measurements, each should run between 8.2 and 10.6 minutes. We can break up some of the larger test cases later into more shardable pieces (in particular: "test"). Update golang/go#10029 Change-Id: Ib3e008b61ce6f28d799ae298c8467bf62d89e5d4 Reviewed-on: https://go-review.googlesource.com/10056 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-14 23:39:58 +03:00
})
dashboard: add linux-arm and linux-arm-arm5 builders Change-Id: Ib8997f76bdf7965d34543cbb6221d6bdb25246fa Reviewed-on: https://go-review.googlesource.com/9840 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2015-05-07 21:36:25 +03:00
addBuilder(BuildConfig{
dashboard: add a make.bash-only builder on real ARM hardware This is a new builder in prep for the change to the "linux-arm" builder where the GOARCH=arm make.bash will be cross-compiled from a Kubernetes container on fast hardware. Updates golang/go#17105 (cross-compile ARM builders' make.bash) Updates golang/go#17104 (5 minute trybots) Change-Id: Icfd2644d77639f731151abe54839322960418254 Reviewed-on: https://go-review.googlesource.com/29670 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 02:29:31 +03:00
Name: "linux-arm-nativemake",
Notes: "runs make.bash on real ARM hardware, but does not run tests",
dashboard: fix build Tests (and binaries) panicked before this. Change-Id: If984e9e30016d403af2a94971fc4a419a269b73a Reviewed-on: https://go-review.googlesource.com/46510 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-23 17:02:38 +03:00
HostType: "host-linux-arm-scaleway",
dashboard: add a make.bash-only builder on real ARM hardware This is a new builder in prep for the change to the "linux-arm" builder where the GOARCH=arm make.bash will be cross-compiled from a Kubernetes container on fast hardware. Updates golang/go#17105 (cross-compile ARM builders' make.bash) Updates golang/go#17104 (5 minute trybots) Change-Id: Icfd2644d77639f731151abe54839322960418254 Reviewed-on: https://go-review.googlesource.com/29670 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 02:29:31 +03:00
StopAfterMake: true,
})
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
addBuilder(BuildConfig{
Name: "linux-arm-arm5spacemonkey",
HostType: "host-linux-arm5spacemonkey",
dashboard: increase arm5 builder timeouts Change-Id: I1c1af4f5f6d98ca78da2ab27c3006da867492777 Reviewed-on: https://go-review.googlesource.com/82935 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-12-08 20:39:57 +03:00
env: []string{
"GOARM=5",
"GO_TEST_TIMEOUT_SCALE=4", // arm is normally 2; double that.
},
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
ShouldRunDistTest: func(distTest string, isTry bool) bool {
if strings.Contains(distTest, "vendor/github.com/google/pprof") {
// Not worth it. And broken.
return false
}
dashboard: skip some slow/broken tests on arm5 builder Updates golang/go#24754 Change-Id: Ie5e94a8fd0b76e6394249a734d8fc1a61650dbad Reviewed-on: https://go-review.googlesource.com/106118 Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2018-04-10 23:03:26 +03:00
if distTest == "api" {
// Broken on this build config (Issue
// 24754), and not worth it on slow
// builder. It's covered by other
// builders anyway.
return false
}
if strings.HasPrefix(distTest, "test:") {
// Slow, and not worth it on slow builder.
return false
}
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
return true
},
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
})
env: update nacl builders to be VM-based Be lazy and use sid as a base, instead of trusty. This replaces my earlier failed attempt: https://golang.org/cl/4960 Fixes golang/go#9879 Change-Id: Ia0a39c99747844b6e34bf7561c01dce49fa3b226 Reviewed-on: https://go-review.googlesource.com/5191 Reviewed-by: Russ Cox <rsc@golang.org>
2015-02-19 01:12:22 +03:00
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "nacl-386",
HostType: "host-nacl-kube",
dashboard: revert accidental inclusion of nacl subrepo trybots Fixes golang/go#28582 Change-Id: Ie3bbb9f4dc1a12c5c4a838165c81f5e2a762a9fe Reviewed-on: https://go-review.googlesource.com/c/147337 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-11-04 04:44:35 +03:00
tryBot: explicitTrySet("go"),
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 2,
numTryTestHelpers: 3,
env: []string{"GOOS=nacl", "GOARCH=386", "GOHOSTOS=linux", "GOHOSTARCH=amd64"},
all: upgrade nacl builders to pepper_44 and Kube Updates golang/go#12379 Change-Id: I40bd6e37953bc509f557ff295374b51de550726c Reviewed-on: https://go-review.googlesource.com/18946 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-01-27 02:13:47 +03:00
})
addBuilder(BuildConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Name: "nacl-amd64p32",
HostType: "host-nacl-kube",
dashboard: revert accidental inclusion of nacl subrepo trybots Fixes golang/go#28582 Change-Id: Ie3bbb9f4dc1a12c5c4a838165c81f5e2a762a9fe Reviewed-on: https://go-review.googlesource.com/c/147337 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-11-04 04:44:35 +03:00
tryBot: explicitTrySet("go"),
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
MaxAtOnce: 2,
numTryTestHelpers: 3,
env: []string{"GOOS=nacl", "GOARCH=amd64p32", "GOHOSTOS=linux", "GOHOSTARCH=amd64"},
all: upgrade nacl builders to pepper_44 and Kube Updates golang/go#12379 Change-Id: I40bd6e37953bc509f557ff295374b51de550726c Reviewed-on: https://go-review.googlesource.com/18946 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-01-27 02:13:47 +03:00
})
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
addBuilder(BuildConfig{
dashboard: speed up js/wasm trybots We don't need to run all the tests in a trybot. Run just enough to get good coverage, without going over 5 minutes. Any regressions elsewhere will be caught by the build.golang.org (slower) runs. Change-Id: I32f1fc17681bfb509844d2bd35b05c950806d283 Reviewed-on: https://go-review.googlesource.com/121938 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-28 23:30:35 +03:00
Name: "js-wasm",
HostType: "host-js-wasm",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("go"),
dashboard: speed up js/wasm trybots We don't need to run all the tests in a trybot. Run just enough to get good coverage, without going over 5 minutes. Any regressions elsewhere will be caught by the build.golang.org (slower) runs. Change-Id: I32f1fc17681bfb509844d2bd35b05c950806d283 Reviewed-on: https://go-review.googlesource.com/121938 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-28 23:30:35 +03:00
ShouldRunDistTest: func(distTest string, isTry bool) bool {
if isTry {
if strings.HasPrefix(distTest, "test:") {
return false
}
if strings.Contains(distTest, "/internal/") ||
strings.Contains(distTest, "vendor/golang.org/x/arch") {
return false
}
switch distTest {
case "cmd/go", "nolibgcc:crypto/x509":
return false
}
return true
}
return true
},
dashboard: enable trybots for js/wasm Fixes golang/go#26015 Change-Id: I25656a8e2918196cf950e29184c783503da9f905 Reviewed-on: https://go-review.googlesource.com/120775 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-06-25 21:12:11 +03:00
numTryTestHelpers: 4,
GoDeps: []string{
"3dced519cbabc213df369d9112206986e62687fa", // first passing commit
},
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
env: []string{
"GOOS=js", "GOARCH=wasm", "GOHOSTOS=linux", "GOHOSTARCH=amd64",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/workdir/go/misc/wasm",
},
})
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
addBuilder(BuildConfig{
dashboard: switch to OpenBSD 6.0 completely, remove OpenBSD 5.8 Updates golang/go#16951 Change-Id: I7b92363671e0ba4c6ab80cef60e7c8600687f440 Reviewed-on: https://go-review.googlesource.com/34384 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-12-15 01:41:25 +03:00
Name: "openbsd-amd64-60",
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
HostType: "host-openbsd-amd64-60",
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
ShouldRunDistTest: noTestDir,
dashboard: disable OpenBSD 6.0 builders Fixes golang/go#24131 Change-Id: I08b014c2b2f06fc1eb6e2b1f31af77529987609f Reviewed-on: https://go-review.googlesource.com/97417 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-02-27 20:26:26 +03:00
TryOnly: true, // disabled by default; Go 1.11+ don't support it anymore
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil,
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 1,
dashboard: add OpenBSD 6.0 amd64 builder, bump mips timeouts Start of OpenBSD 6.0 builders. Not trybots yet, and don't have 386 yet. Waiting on tmpfs resolution first, and then I'll delete the 5.8 builders. Updates golang/go#16951 Change-Id: Ib9c68090d531344125ac5410e732ae0c664a0dc4 Reviewed-on: https://go-review.googlesource.com/34276 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-10 22:10:49 +03:00
numTestHelpers: 2,
numTryTestHelpers: 5,
})
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "openbsd-386-60",
HostType: "host-openbsd-386-60",
ShouldRunDistTest: noTestDir,
dashboard: disable OpenBSD 6.0 builders Fixes golang/go#24131 Change-Id: I08b014c2b2f06fc1eb6e2b1f31af77529987609f Reviewed-on: https://go-review.googlesource.com/97417 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-02-27 20:26:26 +03:00
TryOnly: true, // disabled by default; Go 1.11+ don't support it anymore
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil,
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
MaxAtOnce: 1,
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
env: []string{
// cmd/go takes ~192 seconds on openbsd-386
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
})
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "openbsd-386-62",
HostType: "host-openbsd-386-62",
ShouldRunDistTest: noTestDir,
MaxAtOnce: 1,
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
env: []string{
// cmd/go takes ~192 seconds on openbsd-386
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
})
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
addBuilder(BuildConfig{
Name: "openbsd-amd64-62",
HostType: "host-openbsd-amd64-62",
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
ShouldRunDistTest: noTestDir,
dashboard: switch OpenBSD trybot from 6.2 to 6.4 OpenBSD 6.4 is the current OpenBSD release, while 6.2 has just stopped being supported. The new 6.4 builder has proven to be stable and reliable, so start using it for trybots as well. Looking at the non-trybot build times for the Go repository with OpenBSD 6.2 and 6.4 builders to get a sense of the build speed, it seems OpenBSD 6.4 is about 15% slower (1200~ seconds on 6.2 vs 1380~ seconds on 6.4). This might be due to security-oriented changes in OpenBSD 6.3 and 6.4 that were done to mitigate Spectre/Meltdown vulnerabilities. CL 145022 has helped recover some lost performance by re-enabling SMT. OpenBSD trybots seem to take around 6 minutes now, so there's some room to make them slower before they're the bottleneck. Try to update for now, and if it becomes a bottleneck, then we can prioritize investigating ways to improve OpenBSD 6.4 builder performance further if possible. References: • https://www.openbsd.org/64.html • https://www.openbsd.org/63.html • https://man.openbsd.org/sysctl.2#HW_SMT_2 Fixes golang/go#26833 Change-Id: I42c136fab800df46866a95017a8f70047b37f847 Reviewed-on: https://go-review.googlesource.com/c/144777 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 02:07:13 +03:00
tryBot: nil,
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
numTestHelpers: 0,
numTryTestHelpers: 5,
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 1,
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
})
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
addBuilder(BuildConfig{
Name: "openbsd-amd64-64",
HostType: "host-openbsd-amd64-64",
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
MinimumGoVersion: types.MajorMinor{1, 11},
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
ShouldRunDistTest: noTestDir,
dashboard: switch OpenBSD trybot from 6.2 to 6.4 OpenBSD 6.4 is the current OpenBSD release, while 6.2 has just stopped being supported. The new 6.4 builder has proven to be stable and reliable, so start using it for trybots as well. Looking at the non-trybot build times for the Go repository with OpenBSD 6.2 and 6.4 builders to get a sense of the build speed, it seems OpenBSD 6.4 is about 15% slower (1200~ seconds on 6.2 vs 1380~ seconds on 6.4). This might be due to security-oriented changes in OpenBSD 6.3 and 6.4 that were done to mitigate Spectre/Meltdown vulnerabilities. CL 145022 has helped recover some lost performance by re-enabling SMT. OpenBSD trybots seem to take around 6 minutes now, so there's some room to make them slower before they're the bottleneck. Try to update for now, and if it becomes a bottleneck, then we can prioritize investigating ways to improve OpenBSD 6.4 builder performance further if possible. References: • https://www.openbsd.org/64.html • https://www.openbsd.org/63.html • https://man.openbsd.org/sysctl.2#HW_SMT_2 Fixes golang/go#26833 Change-Id: I42c136fab800df46866a95017a8f70047b37f847 Reviewed-on: https://go-review.googlesource.com/c/144777 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 02:07:13 +03:00
tryBot: defaultTrySet(),
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
numTestHelpers: 0,
numTryTestHelpers: 5,
MaxAtOnce: 1,
})
addBuilder(BuildConfig{
Name: "openbsd-386-64",
HostType: "host-openbsd-386-64",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("sys"),
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
ShouldRunDistTest: noTestDir,
MaxAtOnce: 1,
})
cmd/coordinator/buildongce, dashboard: fix basepin creation, add NetBSD Updates golang/go#8643 Change-Id: I031ffb52537500c52df34f54552296c61dc2df10 Reviewed-on: https://go-review.googlesource.com/21493 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-04-04 21:03:49 +03:00
addBuilder(BuildConfig{
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
Name: "netbsd-amd64-8_0",
HostType: "host-netbsd-amd64-8_0",
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
ShouldRunDistTest: noTestDir,
MaxAtOnce: 1,
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("sys"),
all: enable NetBSD 7.0 builders Fixes golang/go#8643 Change-Id: Id6a99d0bf4f032c5f5a2e5c282103cc98a515c7b Reviewed-on: https://go-review.googlesource.com/37469 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-26 21:11:21 +03:00
})
addBuilder(BuildConfig{
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
Name: "netbsd-386-8_0",
HostType: "host-netbsd-386-8_0",
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
ShouldRunDistTest: noTestDir,
MaxAtOnce: 1,
dashboard: update netbsd builders to 8.0RC1 Rename the 8branch builders from “8branch” to “8_0” (in keeping with the version suffixes for FreeBSD and Darwin), and fix an apparent typo in the 386 SSHUsername field. Updates golang/go#25206. Updates golang/go#25109. Change-Id: I6053c8de17dcb3c7ff32da05c51902ac4bcc45fe Reviewed-on: https://go-review.googlesource.com/110626 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-28 01:47:56 +03:00
// This builder currently hangs in the “../test” phase of all.bash.
// (https://golang.org/issue/25206)
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
TryOnly: true, // Disable regular builds.
tryBot: nil, // Disable trybots.
cmd/coordinator/buildongce, dashboard: fix basepin creation, add NetBSD Updates golang/go#8643 Change-Id: I031ffb52537500c52df34f54552296c61dc2df10 Reviewed-on: https://go-review.googlesource.com/21493 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-04-04 21:03:49 +03:00
})
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "plan9-386",
HostType: "host-plan9-386-gce",
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
numTestHelpers: 1,
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
env/plan9: revert SMP changes, stop running cmd/go tests Revert "env: enable SMP in Plan 9 image" This reverts commit 6478d3e7bdff4c8d810ba44f970ca4b0bfc92405. Revert "dashboard: bump plan9 GCE image" This reverts commit ce208996bf954a87d68789bb6a9000256f016c77. Updates golang/go#29822 Change-Id: I47e75d135e092ee9d5a80c66e91c1048c67cd5f6 Reviewed-on: https://go-review.googlesource.com/c/158542 Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: David du Colombier <0intro@gmail.com>
2019-01-19 01:19:58 +03:00
ShouldRunDistTest: func(distTestName string, isTry bool) bool {
switch distTestName {
case "api",
"go_test:cmd/go": // takes over 20 minutes without working SMP
return false
}
return true
},
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
})
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "windows-amd64-2008",
HostType: "host-windows-amd64-2008",
ShouldRunDistTest: noTestDir,
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
})
cmd/buildlet, dashboard: support for v2 windows images, 386 builder - images now include 32bit gcc in c:\godeps\gcc32 - 64bit gcc has moved to c:\godeps\gcc64 - update host definitions for new image versions - add windows-386-2008 host Updates golang/go#17513 Change-Id: Ib3f5d4b2b1702c05089cf633529ebf38db270a43 Reviewed-on: https://go-review.googlesource.com/42014 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-27 23:12:08 +03:00
addBuilder(BuildConfig{
Name: "windows-386-2008",
HostType: "host-windows-amd64-2008",
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
ShouldRunDistTest: fasterTrybots,
cmd/buildlet, dashboard: support for v2 windows images, 386 builder - images now include 32bit gcc in c:\godeps\gcc32 - 64bit gcc has moved to c:\godeps\gcc64 - update host definitions for new image versions - add windows-386-2008 host Updates golang/go#17513 Change-Id: Ib3f5d4b2b1702c05089cf633529ebf38db270a43 Reviewed-on: https://go-review.googlesource.com/42014 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-27 23:12:08 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
dashboard: restart netbsd-386 builder Also clean up some redundant comments, move trybots from OpenBSD 6.0 to 6.2, and add some missing MaxAtOnce caps. Updates golang/go#20852 Change-Id: I3d862ca7a7172952ec2254b583e4bf6102b026e6 Reviewed-on: https://go-review.googlesource.com/80876 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-30 01:43:26 +03:00
MaxAtOnce: 2,
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
numTryTestHelpers: 4,
cmd/buildlet, dashboard: support for v2 windows images, 386 builder - images now include 32bit gcc in c:\godeps\gcc32 - 64bit gcc has moved to c:\godeps\gcc64 - update host definitions for new image versions - add windows-386-2008 host Updates golang/go#17513 Change-Id: Ib3f5d4b2b1702c05089cf633529ebf38db270a43 Reviewed-on: https://go-review.googlesource.com/42014 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-27 23:12:08 +03:00
})
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "windows-amd64-2012",
HostType: "host-windows-amd64-2012",
ShouldRunDistTest: noTestDir,
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
MaxAtOnce: 2,
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
})
build: use buildlet on Windows Also adds "gomote rmall" command. Fixes golang/go#8640 Fixes golang/go#9799 Change-Id: I41b62b16df5e855e1367a9cc1c370bb1f928c242 Reviewed-on: https://go-review.googlesource.com/4132 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2015-02-07 04:32:15 +03:00
addBuilder(BuildConfig{
dashboard: update Windows builders, remove old manual Windows builder And clean up some other builders. Change-Id: I1f71836da7008e58d3e76e2cc3170e96cd57ddf6 Reviewed-on: https://go-review.googlesource.com/51432 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-27 07:52:12 +03:00
Name: "windows-amd64-2016",
HostType: "host-windows-amd64-2016",
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
ShouldRunDistTest: fasterTrybots,
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
dashboard: update Windows builders, remove old manual Windows builder And clean up some other builders. Change-Id: I1f71836da7008e58d3e76e2cc3170e96cd57ddf6 Reviewed-on: https://go-review.googlesource.com/51432 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-27 07:52:12 +03:00
numTryTestHelpers: 5,
build: use buildlet on Windows Also adds "gomote rmall" command. Fixes golang/go#8640 Fixes golang/go#9799 Change-Id: I41b62b16df5e855e1367a9cc1c370bb1f928c242 Reviewed-on: https://go-review.googlesource.com/4132 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2015-02-07 04:32:15 +03:00
})
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "windows-amd64-race",
dashboard: retire windows-amd64-gce builder, replace with windows-amd64-2008 The windows-amd64-2008 is the same OS as windows-amd64-gce but is an auto-generated image. TODO: 386 auto-generated Windows builders, and then maybe we'll move TryBots to Windows Server 2016. One step at a time. This should be a no-op. I verified performance is the same. Updates golang/go#17513 Change-Id: I34984db14b87d03771e15465978b1687df6895f7 Reviewed-on: https://go-review.googlesource.com/41611 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-24 23:32:02 +03:00
HostType: "host-windows-amd64-2008",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "Only runs -race tests (./race.bat)",
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2"},
build: use buildlet on Windows Also adds "gomote rmall" command. Fixes golang/go#8640 Fixes golang/go#9799 Change-Id: I41b62b16df5e855e1367a9cc1c370bb1f928c242 Reviewed-on: https://go-review.googlesource.com/4132 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2015-02-07 04:32:15 +03:00
})
dashboard: add windows-arm builder Updates golang/go#26148 Change-Id: I23a2c7bd1c2f8f7c960aaedfe0dcdee4e11825ab Reviewed-on: https://go-review.googlesource.com/125643 Reviewed-by: Andrew Bonventre <andybons@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-24 22:59:34 +03:00
addBuilder(BuildConfig{
dashboard: skip writing snapshot for windows/arm builder Writing the snapshot to GCS is timing out for the Windows/ARM builder. It tars about 500MB of data, resulting in about 175MB of data, which it then sends to GCS. It takes about 4:30 to tar the workdir/go directory with the built-in tar command, leaving only 30s to transfer 175MB across the web, so it's not surprising that it times out. Updates golang/go#26148 Change-Id: I7e70aeaf1ac6334c17fdca64ae9ab946c3d87d9a Reviewed-on: https://go-review.googlesource.com/c/153518 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-12-11 03:56:05 +03:00
Name: "windows-arm",
HostType: "host-windows-arm-iotcore",
SkipSnapshot: true,
dashboard: add windows-arm builder Updates golang/go#26148 Change-Id: I23a2c7bd1c2f8f7c960aaedfe0dcdee4e11825ab Reviewed-on: https://go-review.googlesource.com/125643 Reviewed-by: Andrew Bonventre <andybons@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-24 22:59:34 +03:00
env: []string{
"GOARM=7",
"GO_TEST_TIMEOUT_SCALE=2",
},
})
dashboard: add OS X 10.8 and 10.11 reverse builders Updates golang/go#9495 Updates golang/go#12185 Change-Id: I84548011823334f0876636f8e9685e07238cafc4 Reviewed-on: https://go-review.googlesource.com/28171 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 22:55:08 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "darwin-amd64-10_8",
HostType: "host-darwin-10_8",
ShouldRunDistTest: noTestDir,
dashboard, app/appengine: disable macOS 10.8 builders, disable Go 1.8 builders Change-Id: I84532de81c9eaaf3a989d1f855053c6104e2195d Reviewed-on: https://go-review.googlesource.com/95555 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-02-20 22:36:16 +03:00
TryOnly: true, // but not in trybot set, so effectively disabled
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil,
dashboard: add OS X 10.8 and 10.11 reverse builders Updates golang/go#9495 Updates golang/go#12185 Change-Id: I84548011823334f0876636f8e9685e07238cafc4 Reviewed-on: https://go-review.googlesource.com/28171 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 22:55:08 +03:00
})
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "darwin-amd64-10_10",
HostType: "host-darwin-10_10",
ShouldRunDistTest: noTestDir,
dashboard: add OS X 10.8 and 10.11 reverse builders Updates golang/go#9495 Updates golang/go#12185 Change-Id: I84548011823334f0876636f8e9685e07238cafc4 Reviewed-on: https://go-review.googlesource.com/28171 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 22:55:08 +03:00
})
cmd/coordinator, dashboard: switch to new VMWare-based OS X builders Ignore the old darwin-{amd64,386}-10_10 builders. Don't give them an error, but pretend they don't exist. Also: switch trybots from OS X 10.10 to OS X 10.11, and re-enable sharding. Let's hope for the best. See golang/go#12979. This also enables subrepo tests for all OS X versions. darwin-386-* is currently offline, pending some golang/go#17009 Updates golang/go#9495 (OS X virtualization) Change-Id: I4d53a79087404b5e8051d1aff0c668a92625f442 Reviewed-on: https://go-review.googlesource.com/28583 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-07 08:41:02 +03:00
addBuilder(BuildConfig{
dashboard: fix typo from CL 94601 Sigh. Change-Id: Ife6006f9b0c65fa6ee7be50b6c9f55319e0fa28d Reviewed-on: https://go-review.googlesource.com/94602 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-02-16 02:43:31 +03:00
Name: "darwin-amd64-10_11",
HostType: "host-darwin-10_11",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil, // disabled until Macs fixed; https://golang.org/issue/23859
dashboard: fix typo from CL 94601 Sigh. Change-Id: Ife6006f9b0c65fa6ee7be50b6c9f55319e0fa28d Reviewed-on: https://go-review.googlesource.com/94602 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-02-16 02:43:31 +03:00
ShouldRunDistTest: noTestDir,
numTryTestHelpers: 3,
cmd/coordinator, dashboard: switch to new VMWare-based OS X builders Ignore the old darwin-{amd64,386}-10_10 builders. Don't give them an error, but pretend they don't exist. Also: switch trybots from OS X 10.10 to OS X 10.11, and re-enable sharding. Let's hope for the best. See golang/go#12979. This also enables subrepo tests for all OS X versions. darwin-386-* is currently offline, pending some golang/go#17009 Updates golang/go#9495 (OS X virtualization) Change-Id: I4d53a79087404b5e8051d1aff0c668a92625f442 Reviewed-on: https://go-review.googlesource.com/28583 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-07 08:41:02 +03:00
})
dashboard: restore darwin-386 buidler Fixes golang/go#17009 Change-Id: Ia9008bcb7b47b7671c158c1a00afc4e765805e4e Reviewed-on: https://go-review.googlesource.com/41617 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-25 02:03:42 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "darwin-386-10_11",
HostType: "host-darwin-10_11",
ShouldRunDistTest: noTestDir,
MaxAtOnce: 1,
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
})
addBuilder(BuildConfig{
Name: "darwin-amd64-10_12",
HostType: "host-darwin-10_12",
ShouldRunDistTest: noTestDir,
dashboard: restore darwin-386 buidler Fixes golang/go#17009 Change-Id: Ia9008bcb7b47b7671c158c1a00afc4e765805e4e Reviewed-on: https://go-review.googlesource.com/41617 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-25 02:03:42 +03:00
})
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "darwin-amd64-race",
HostType: "host-darwin-10_11",
ShouldRunDistTest: noTestDir,
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
})
dashboard: add new iOS builders Change-Id: I2e09ea66759bcb8ccd93912a410059638a7f800d Reviewed-on: https://go-review.googlesource.com/36095 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2017-02-02 18:00:21 +03:00
addBuilder(BuildConfig{
dashboard: add iphone 5 device for darwin/arm builds The newly restored iPhone 6 device now runs iOS 11 which can't run arm binaries. Add another device (iPhone 5) for darwin/arm builds and use GOIOS_DEVICE_ID to distinguish between them. While we're here, update the model numbers to reflect reality. Change-Id: Iaef33469d19b99354d0dd72577d31289d179c82c Reviewed-on: https://go-review.googlesource.com/67090 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2017-09-29 15:10:42 +03:00
Name: "darwin-arm-a1428ios",
dashboard: undo ios builder split Running multiple iOS devices on a single host was too ambitious for the brittle test harness. This CL reverts most of the split from CL 57297 except the device model names are updated to reflect reality. I'm sorry. Change-Id: I4e9b1d677eb7c57178f112dedb62580fc74a461d Reviewed-on: https://go-review.googlesource.com/59510 Reviewed-by: Sarah Adams <shadams@google.com>
2017-08-28 20:11:57 +03:00
HostType: "host-darwin-amd64-eliasnaur-ios",
dashboard: switch iOS hardware Jakob Borg kindly supplied a new iPhone 6 to replace the ailing iPhone 5 (arm) and iPhone 6 (arm64) that currenctly runs the iOS builders. The new phone runs iOS 10 that can run both arm and arm64, so instead of replacing the device ids specified with GOIOS_DEVICE_ID, simply remove them altogether. Change-Id: I36cf574a902b953a3d2bf2a5de50741dff1b940d Reviewed-on: https://go-review.googlesource.com/119415 Run-TryBot: Elias Naur <elias.naur@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-18 16:50:26 +03:00
Notes: "iPhone 6 (model A1586), via a Mac Mini; owned by elias.naur",
env: []string{"GOARCH=arm"},
dashboard: add new iOS builders Change-Id: I2e09ea66759bcb8ccd93912a410059638a7f800d Reviewed-on: https://go-review.googlesource.com/36095 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2017-02-02 18:00:21 +03:00
})
addBuilder(BuildConfig{
dashboard: add iphone 5 device for darwin/arm builds The newly restored iPhone 6 device now runs iOS 11 which can't run arm binaries. Add another device (iPhone 5) for darwin/arm builds and use GOIOS_DEVICE_ID to distinguish between them. While we're here, update the model numbers to reflect reality. Change-Id: Iaef33469d19b99354d0dd72577d31289d179c82c Reviewed-on: https://go-review.googlesource.com/67090 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2017-09-29 15:10:42 +03:00
Name: "darwin-arm64-a1549ios",
dashboard: undo ios builder split Running multiple iOS devices on a single host was too ambitious for the brittle test harness. This CL reverts most of the split from CL 57297 except the device model names are updated to reflect reality. I'm sorry. Change-Id: I4e9b1d677eb7c57178f112dedb62580fc74a461d Reviewed-on: https://go-review.googlesource.com/59510 Reviewed-by: Sarah Adams <shadams@google.com>
2017-08-28 20:11:57 +03:00
HostType: "host-darwin-amd64-eliasnaur-ios",
dashboard: switch iOS hardware Jakob Borg kindly supplied a new iPhone 6 to replace the ailing iPhone 5 (arm) and iPhone 6 (arm64) that currenctly runs the iOS builders. The new phone runs iOS 10 that can run both arm and arm64, so instead of replacing the device ids specified with GOIOS_DEVICE_ID, simply remove them altogether. Change-Id: I36cf574a902b953a3d2bf2a5de50741dff1b940d Reviewed-on: https://go-review.googlesource.com/119415 Run-TryBot: Elias Naur <elias.naur@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-18 16:50:26 +03:00
Notes: "iPhone 6 (model A1586), via a Mac Mini; owned by elias.naur",
env: []string{"GOARCH=arm64"},
dashboard: add new iOS builders Change-Id: I2e09ea66759bcb8ccd93912a410059638a7f800d Reviewed-on: https://go-review.googlesource.com/36095 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2017-02-02 18:00:21 +03:00
})
dashboard: add zenly as builder for ios (Iphone 5C, 7+) Change-Id: Ie1dd5c53ed0b2d207297ee0cd3ba46a111167e30 Reviewed-on: https://go-review.googlesource.com/125815 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-25 16:14:32 +03:00
addBuilder(BuildConfig{
Name: "darwin-arm-mg912baios",
HostType: "host-darwin-amd64-zenly-ios",
Notes: "iPhone 5C (model MG912B/A), via a MacBook Pro; owned by zenly",
env: []string{
"GOARCH=arm",
"GOIOS_DEVICE_ID=8e5c23a5d0843d1ffe164ea0b2f2500599c3ebff",
},
})
addBuilder(BuildConfig{
Name: "darwin-arm64-mn4m2zdaios",
HostType: "host-darwin-amd64-zenly-ios",
Notes: "iPhone 7+ (model MN4M2ZD/A), via a MacBook Pro; owned by zenly",
env: []string{
"GOARCH=arm64",
"GOIOS_DEVICE_ID=5ec20fafe317e1c8ff51efc6d508cf19808474a2",
},
})
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
addBuilder(BuildConfig{
Name: "android-arm-wiko-fever",
dashboard: split mobile builders into iOS and Android Even though both mobile builders will run off the same host Mac Mini, Android and iOS builds can run simultaneously. Split the eliasnaur host in two to allow that. Change-Id: If3db4dadb56b8a4b212dcbb3015e3e6c1b69608d Reviewed-on: https://go-review.googlesource.com/39130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-31 11:08:01 +03:00
HostType: "host-darwin-amd64-eliasnaur-android",
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
Notes: "Android Wiko Fever phone running Android 6.0, via a Mac Mini",
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
PostSubmitBuildPolicy: func(repo string) bool {
return repo == "go" || repo == "mobile"
},
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
env: []string{
"GOARCH=arm",
dashboard: remove eliasnaur Android emulators Now that we hace Android emulator builders on GCE, drop the old ones running on hardware. Also update the configuration of the remaining builders now that androidtest.bash is no longer necessary. Change-Id: I09cab08b28952f967205d6c35f36eb6e05a17950 Reviewed-on: https://go-review.googlesource.com/c/164397 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 10:19:36 +03:00
"GOOS=android",
"GOHOSTARCH=amd64",
"GOHOSTOS=linux",
"CGO_ENABLED=1",
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
},
})
addBuilder(BuildConfig{
Name: "android-arm64-wiko-fever",
dashboard: split mobile builders into iOS and Android Even though both mobile builders will run off the same host Mac Mini, Android and iOS builds can run simultaneously. Split the eliasnaur host in two to allow that. Change-Id: If3db4dadb56b8a4b212dcbb3015e3e6c1b69608d Reviewed-on: https://go-review.googlesource.com/39130 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-31 11:08:01 +03:00
HostType: "host-darwin-amd64-eliasnaur-android",
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
Notes: "Android Wiko Fever phone running Android 6.0, via a Mac Mini",
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
PostSubmitBuildPolicy: func(repo string) bool {
return repo == "go" || repo == "mobile"
},
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
env: []string{
"GOARCH=arm64",
dashboard: remove eliasnaur Android emulators Now that we hace Android emulator builders on GCE, drop the old ones running on hardware. Also update the configuration of the remaining builders now that androidtest.bash is no longer necessary. Change-Id: I09cab08b28952f967205d6c35f36eb6e05a17950 Reviewed-on: https://go-review.googlesource.com/c/164397 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 10:19:36 +03:00
"GOOS=android",
"GOHOSTARCH=amd64",
"GOHOSTOS=linux",
"CGO_ENABLED=1",
dashboard: add new Android arm and arm64 builders Change-Id: I4b3bceef32c1c29b0885872a7f8f8027cd0b34b2 Reviewed-on: https://go-review.googlesource.com/38831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-30 00:23:59 +03:00
},
})
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
addBuilder(BuildConfig{
Name: "android-386-emu",
HostType: "host-android-amd64-emu", // same amd64 host is used for 386 builder
Notes: "Android emulator on GCE",
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
PostSubmitBuildPolicy: func(repo string) bool {
switch repo {
case "blog", "talks", "review", "tour":
return false
}
return true
},
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
env: []string{
"GOARCH=386",
"GOOS=android",
"GOHOSTARCH=amd64",
"GOHOSTOS=linux",
"CGO_ENABLED=1",
},
})
addBuilder(BuildConfig{
Name: "android-amd64-emu",
HostType: "host-android-amd64-emu",
Notes: "Android emulator on GCE",
cmd/coordinator: test x/mobile on android emu builders; and exp on some Updates golang/go#23824 Change-Id: Iee70bcbfef6a271fdd26694d848a703a200dc0c0 Reviewed-on: https://go-review.googlesource.com/c/164478 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 20:32:45 +03:00
tryBot: func(proj string) bool {
// Only for mobile repo for now, not "go":
return proj == "mobile"
},
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
PostSubmitBuildPolicy: func(repo string) bool {
switch repo {
case "blog", "talks", "review", "tour":
return false
}
return true
},
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
env: []string{
"GOARCH=amd64",
"GOOS=android",
"GOHOSTARCH=amd64",
"GOHOSTOS=linux",
"CGO_ENABLED=1",
},
})
dashboard: add android/386, android/amd64 builders Add an android/386 and an android/amd64 builder, both running on an emulator. Use the new GOANDROID_ADB_FLAGS to direct the build to either a device (android/arm, android/arm64) or an emulator (android/386, android/amd64). Change-Id: I8e25aa5efcbbb186c71c4e108a37ae11b0246c98 Reviewed-on: https://go-review.googlesource.com/88775 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-01-20 20:47:20 +03:00
addBuilder(BuildConfig{
dashboard: add Oracle Solaris development builder Fixes golang/go#15072 Change-Id: I85677636a0f7f8b6a7b2405bb0669c39cc1da7f0 Reviewed-on: https://go-review.googlesource.com/46831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-27 20:11:15 +03:00
Name: "solaris-amd64-oracledev",
HostType: "host-solaris-oracle-shawn",
Notes: "Oracle Solaris development version",
})
x/build: add Solaris amd64 release builder to dashboard Fixes golang/go#20949 Change-Id: I4f85ee76783a1240a2ebd4a2a1f167ab849fcebe Reviewed-on: https://go-review.googlesource.com/47811 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-07 23:41:20 +03:00
addBuilder(BuildConfig{
Name: "solaris-amd64-oraclerel",
HostType: "host-solaris-oracle-amd64-oraclerel",
Notes: "Oracle Solaris release version",
})
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "solaris-amd64-smartosbuildlet",
HostType: "host-solaris-amd64",
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
})
all: add ppc64 (big endian) builders Change-Id: I0fddda33179555a50b6f03dcff13338544b742c3 Reviewed-on: https://go-review.googlesource.com/30599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-06 23:30:58 +03:00
addBuilder(BuildConfig{
Name: "linux-ppc64-buildlet",
HostType: "host-linux-ppc64-osu",
FlakyNet: true,
})
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "linux-ppc64le-buildlet",
HostType: "host-linux-ppc64le-osu",
FlakyNet: true,
dashboard: add arm64 buildlet, update ppc64 buildlet definition Change-Id: I25c6de0f97c3338f9d32f4125b2c7c68c3f3c13c Reviewed-on: https://go-review.googlesource.com/21656 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2016-04-07 22:27:15 +03:00
})
dashboard: add linux-ppc64le POWER9 builder This change adds an extra linux-ppc64le builder running on POWER9. Fixes golang/go#30138 Change-Id: I23bdfc116bb89c6d96be330c2c87827c47705b6b Reviewed-on: https://go-review.googlesource.com/c/163757 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:05:12 +03:00
addBuilder(BuildConfig{
Name: "linux-ppc64le-power9osu",
HostType: "host-linux-ppc64le-power9-osu",
FlakyNet: true,
})
dashboard: add arm64 buildlet, update ppc64 buildlet definition Change-Id: I25c6de0f97c3338f9d32f4125b2c7c68c3f3c13c Reviewed-on: https://go-review.googlesource.com/21656 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2016-04-07 22:27:15 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "linux-arm64-buildlet",
HostType: "host-linux-arm64-linaro",
FlakyNet: true,
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
})
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
addBuilder(BuildConfig{
Name: "linux-arm64-packet",
HostType: "host-linux-arm64-packet",
FlakyNet: true, // unknown; just copied from the linaro one
})
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
addBuilder(BuildConfig{
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
Name: "linux-mips",
HostType: "host-linux-mips",
SkipSnapshot: true,
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
})
addBuilder(BuildConfig{
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
Name: "linux-mipsle",
HostType: "host-linux-mipsle",
SkipSnapshot: true,
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
})
addBuilder(BuildConfig{
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
Name: "linux-mips64",
HostType: "host-linux-mips64",
SkipSnapshot: true,
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
})
addBuilder(BuildConfig{
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
Name: "linux-mips64le",
HostType: "host-linux-mips64le",
SkipSnapshot: true,
all: add linux-mips{,64}{,le} builders Change-Id: I28814c70872cce57f05120073a7d653a6879841c Reviewed-on: https://go-review.googlesource.com/32675 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-11-03 20:35:15 +03:00
})
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
addBuilder(BuildConfig{
Name: "linux-s390x-ibm",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
HostType: "host-linux-s390x",
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
numTestHelpers: 0,
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
})
dashboard: add s390x cross-compile-only builder and release config Also modifies cmd/release to wire it up. Updates golang/go#16443 Change-Id: Ie80a224957f6f662d0aec0fd083e2a57e30436c4 Reviewed-on: https://go-review.googlesource.com/28172 Reviewed-by: Jessica Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 23:32:33 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "linux-s390x-crosscompile",
HostType: "host-s390x-cross-kube",
Notes: "s390x cross-compile builder for releases; doesn't run tests",
CompileOnly: true,
TryOnly: true, // but not in trybot set for now
dashboard: add s390x cross-compile-only builder and release config Also modifies cmd/release to wire it up. Updates golang/go#16443 Change-Id: Ie80a224957f6f662d0aec0fd083e2a57e30436c4 Reviewed-on: https://go-review.googlesource.com/28172 Reviewed-by: Jessica Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 23:32:33 +03:00
env: []string{
"CGO_ENABLED=1",
"GOARCH=s390x",
"GOHOSTARCH=amd64",
"CC_FOR_TARGET=s390x-linux-gnu-gcc",
},
})
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-localdev",
HostType: "host-linux-amd64-localdev",
Notes: "for localhost development only",
TryOnly: true,
})
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
Name: "dragonfly-amd64",
HostType: "host-dragonfly-amd64-tdfbsd",
ShouldRunDistTest: noTestDir,
SkipSnapshot: true,
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
})
addBuilder(BuildConfig{
dashboard: rename freebsd-arm to freebsd-arm-paulzhol I didn't realize that the freebsd-arm builder was actually called freebsd-arm-paulzhol before and I mistakenly created the replacement buildet-based builder as "freebsd-arm". Name it how it was. Change-Id: Ib7d73947ce599c82c628ab323001213e1b4ee20b Reviewed-on: https://go-review.googlesource.com/53358 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 06:54:04 +03:00
Name: "freebsd-arm-paulzhol",
dashboard: set SkipSnapshot on freebsd-arm and dragonfly-amd64 No point snapshotting for non-sharded builders. And snapshotting is too slow and hits the timeout on freebsd-arm. Updates golang/go#21191 Change-Id: I225aa6af493b92aa45589491c69853e13d513dd1 Reviewed-on: https://go-review.googlesource.com/53493 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 02:58:12 +03:00
HostType: "host-freebsd-arm-paulzhol",
SkipSnapshot: true,
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
env: []string{
"GOARM=7",
"CGO_ENABLED=1",
},
})
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
addBuilder(BuildConfig{
Name: "plan9-arm",
HostType: "host-plan9-arm-0intro",
SkipSnapshot: true,
})
addBuilder(BuildConfig{
Name: "nacl-arm",
HostType: "host-nacl-arm-davecheney",
SkipSnapshot: true,
})
addBuilder(BuildConfig{
Name: "plan9-amd64-9front",
HostType: "host-plan9-amd64-0intro",
SkipSnapshot: true,
})
dashboard: add aix-ppc64 builder This is a starting point for adding an aix-ppc64 reverse builder. There can be further adjustments made to it as needed. Updates golang/go#27160. Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508 Reviewed-on: https://go-review.googlesource.com/131335 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-24 21:41:05 +03:00
addBuilder(BuildConfig{
Name: "aix-ppc64",
HostType: "host-aix-ppc64-osuosl",
dashboard: add /opt/freeware/bin to PATH on aix builder Updates golang/go#29065 Fixes golang/go#29078 Change-Id: Ifa9355c9dc988a460b6198913431647ec2c5e6ac Reviewed-on: https://go-review.googlesource.com/c/152457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-12-04 17:46:08 +03:00
env: []string{
"PATH=/opt/freeware/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin",
},
dashboard: add aix-ppc64 builder This is a starting point for adding an aix-ppc64 reverse builder. There can be further adjustments made to it as needed. Updates golang/go#27160. Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508 Reviewed-on: https://go-review.googlesource.com/131335 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-24 21:41:05 +03:00
})
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
}
dashboard: add s390x cross-compile-only builder and release config Also modifies cmd/release to wire it up. Updates golang/go#16443 Change-Id: Ie80a224957f6f662d0aec0fd083e2a57e30436c4 Reviewed-on: https://go-review.googlesource.com/28172 Reviewed-by: Jessica Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 23:32:33 +03:00
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
// addBuilder adds c to the Builders map after doing some sanity
// checks.
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
func addBuilder(c BuildConfig) {
if c.Name == "" {
panic("empty name")
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
if c.HostType == "" {
panic(fmt.Sprintf("missing HostType for builder %q", c.Name))
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
if _, dup := Builders[c.Name]; dup {
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
panic("dup name " + c.Name)
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
if _, ok := Hosts[c.HostType]; !ok {
panic(fmt.Sprintf("undefined HostType %q for builder %q", c.HostType, c.Name))
cmd/coordinator: start of a Kubernetes buildlet pool Updates golang/go#12546 Change-Id: Ia70baeb1df441509f70c77e7b94e7eb9fddccbf3 Reviewed-on: https://go-review.googlesource.com/14392 Reviewed-by: Evan Brown <evanbrown@google.com>
2015-09-09 01:18:47 +03:00
}
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
if c.SkipSnapshot && (c.numTestHelpers > 0 || c.numTryTestHelpers > 0) {
panic(fmt.Sprintf("config %q's SkipSnapshot is not compatible with sharded test helpers", c.Name))
}
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
types := 0
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
for _, fn := range []func() bool{c.IsReverse, c.IsContainer, c.IsVM} {
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
if fn() {
types++
}
}
if types != 1 {
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
panic(fmt.Sprintf("build config %q host type inconsistent (must be Reverse, Image, or VM)", c.Name))
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
}
all: change dashboard.BuildConfig to be a pointer type, like HostConfig BuildConfig grew way too large and has too many slices & such to be a value type. Make it a pointer type, which matches HostConfig. Change-Id: Ie625bece9d6d8c1ec6cff26e77416bd9b1f256d8 Reviewed-on: https://go-review.googlesource.com/c/145077 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 20:52:25 +03:00
Builders[c.Name] = &c
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
}
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
// fasterTrybots is a ShouldRunDistTest policy function.
// It skips (returns false) the test/ directory tests for trybots.
func fasterTrybots(distTest string, isTry bool) bool {
if isTry && strings.HasPrefix(distTest, "test:") {
return false // skip test
}
return true
}
// noTestDir is a ShouldRunDistTest policy function.
// It skips (returns false) the test/ directory tests for all builds.
func noTestDir(distTest string, isTry bool) bool {
if strings.HasPrefix(distTest, "test:") {
return false // skip test
}
return true
}
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
// TryBuildersForProject returns the builders that should run as part of
// a TryBot set for the given project.
// The project argument is of the form "go", "net", "sys", etc.
func TryBuildersForProject(proj string) []*BuildConfig {
var confs []*BuildConfig
for _, conf := range Builders {
if conf.tryBot != nil && conf.tryBot(proj) {
confs = append(confs, conf)
}
}
sort.Slice(confs, func(i, j int) bool {
return confs[i].Name < confs[j].Name
})
return confs
}