go
/
build
зеркало из https://github.com/golang/build.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
build/dashboard/builders.go

2715 строки
92 KiB
Go
Исходник Обычный вид История

dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
// Copyright 2015 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package dashboard contains shared configuration and logic used by various
// pieces of the Go continuous build system.
package dashboard
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
import (
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"fmt"
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
"os"
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
"sort"
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
"strconv"
"strings"
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
"time"
all: consolidate configuration for coordinator and gomote buildenv.Environment type defines configuration options: - Coordinator uses the GCE project name to lookup config. A custom config name can be provided at runtime to override. - The conventional prod and stage project names ('symbolic-datum-552' and 'go-dashboard-dev') map to prod and staging configuration structs. - Production and staging status is explicitly defined in configuration. - GCS bucket names for buildlet, logs, and snapshots are configurable. Change-Id: I7e6d7874eb0bdfe35dbdd5fcf6212ab50d576b88 Reviewed-on: https://go-review.googlesource.com/19502 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-15 02:59:59 +03:00
"golang.org/x/build/buildenv"
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
"golang.org/x/build/maintner/maintnerd/maintapi/version"
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
"golang.org/x/build/types"
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
)
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
// slowBotAliases maps short names from TRY= comments to which builder to run.
//
// TODO: we'll likely expand this, or move it, or change the matching
// syntax entirely. This is a first draft.
var slowBotAliases = map[string]string{
// Known missing builders:
dashboard: add openbsd-arm64-jsing builder (reverse) Update golang/go#31656 Fixes golang/go#34035 Change-Id: I7e3d20be148ad5111efeb78702352da14274938e Reviewed-on: https://go-review.googlesource.com/c/build/+/250117 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-20 21:42:01 +03:00
"darwin-arm": "", // TODO(golang.org/issue/37611): Remove once port is removed.
"darwin-arm64": "", // TODO(golang.org/issue/39782): Add builder for darwin/arm64.
dashboard: skip ios-amd64 SlowBot alias It's known there is no builder for it yet. Skip it so that TestSlowBotAliases passes. For golang/go#42177. Updates golang/go#42100. Change-Id: I5f1a6b17e6aa4b57cbdea44f2eeb1d4817e5f67f Reviewed-on: https://go-review.googlesource.com/c/build/+/264738 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-10-24 01:02:06 +03:00
"ios-amd64": "", // There is no builder for the iOS Simulator. See issues 42100 and 42177.
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"386": "linux-386",
"aix": "aix-ppc64",
"amd64": "linux-amd64",
"android": "android-amd64-emu",
"android-386": "android-386-emu",
"android-amd64": "android-amd64-emu",
"android-arm": "android-arm-corellium",
"android-arm64": "android-arm64-corellium",
"arm": "linux-arm",
"arm64": "linux-arm64-packet",
"darwin": "darwin-amd64-10_14",
dashboard: disable darwin/386 builder on master The 32-bit pairs darwin/386 and darwin/arm are removed in Go 1.15, and the master branch is where Go 1.15 is being developed. The builder has started to fail after CL 226985 (as intended), so remove it. Keep it running on Go 1.14 and 1.13 release branches. Also update the misc-compile-darwin builder description. For golang/go#37610. For golang/go#37611. Change-Id: Ia70386c5ae2821b4397048dcb0862a066463d6de Reviewed-on: https://go-review.googlesource.com/c/build/+/227181 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2020-04-04 06:21:35 +03:00
"darwin-386": "darwin-386-10_14", // TODO(golang.org/issue/37610): Remove when Go 1.14 is no longer supported.
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"darwin-amd64": "darwin-amd64-10_14",
dashboard,env: move iOS builders to GOOS=ios Updates golang/go#41610 Change-Id: Ie8cb3294da5566e09fc06c01e679b8b773a439cf Reviewed-on: https://go-review.googlesource.com/c/build/+/259358 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Go Bot <gobot@golang.org> Trust: Elias Naur <mail@eliasnaur.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-10-03 12:37:07 +03:00
"ios-arm64": "ios-arm64-corellium",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"dragonfly": "dragonfly-amd64",
"freebsd": "freebsd-amd64-12_0",
"freebsd-386": "freebsd-386-12_0",
"freebsd-amd64": "freebsd-amd64-12_0",
"freebsd-arm": "freebsd-arm-paulzhol",
dashboard: add SlowBot alias for freebsd-arm64-dmgk This should fix TestSlowBotAliases on tip. Updates golang/go#24715 Updates golang/go#34501 Change-Id: I80f99a5c24ef0860717605816a753fe6625aff17 Reviewed-on: https://go-review.googlesource.com/c/build/+/203421 Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-25 22:28:26 +03:00
"freebsd-arm64": "freebsd-arm64-dmgk",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"illumos": "illumos-amd64",
dashboard,env: move iOS builders to GOOS=ios Updates golang/go#41610 Change-Id: Ie8cb3294da5566e09fc06c01e679b8b773a439cf Reviewed-on: https://go-review.googlesource.com/c/build/+/259358 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Go Bot <gobot@golang.org> Trust: Elias Naur <mail@eliasnaur.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-10-03 12:37:07 +03:00
"ios": "ios-arm64-corellium",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"js": "js-wasm",
"linux": "linux-amd64",
"linux-arm64": "linux-arm64-packet",
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
"linux-mips": "linux-mips-rtrk",
"linux-mips64": "linux-mips64-rtrk",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"linux-mips64le": "linux-mips64le-mengzhuo",
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
"linux-mipsle": "linux-mipsle-rtrk",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"linux-ppc64": "linux-ppc64-buildlet",
"linux-ppc64le": "linux-ppc64le-buildlet",
dashboard: add linux-riscv64-jsing builder (reverse) Also use this as the riscv64 slowbot, since linux-riscv64-unleashed has been MIA for quite some time. For golang/go#27532 Change-Id: I01309a66b96d4c35dbc9d1e08f55f5357b0acba0 Reviewed-on: https://go-review.googlesource.com/c/build/+/257917 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-09-27 21:25:41 +03:00
"linux-riscv64": "linux-riscv64-jsing",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"linux-s390x": "linux-s390x-ibm",
dashboard: add 'longtest' SlowBot alias Change-Id: I83cda2cf521b537f4c879d39b473d48faab53965 Reviewed-on: https://go-review.googlesource.com/c/build/+/205858 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-07 20:47:44 +03:00
"longtest": "linux-amd64-longtest",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"mac": "darwin-amd64-10_14",
"macos": "darwin-amd64-10_14",
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
"mips": "linux-mips-rtrk",
"mips64": "linux-mips64-rtrk",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"mips64le": "linux-mips64le-mengzhuo",
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
"mipsle": "linux-mipsle-rtrk",
dashboard: remove netbsd-{amd64,386}-8_0 builders They've been superseded by the netbsd-{amd64,386}-9_0 builders. Fixes golang/go#37244. For golang/go#31726. Change-Id: I17001d250b01d885ecd141b7bc07fce3de73e9a1 Reviewed-on: https://go-review.googlesource.com/c/build/+/227738 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Benny Siegert <bsiegert@gmail.com>
2020-04-09 14:59:16 +03:00
"netbsd": "netbsd-amd64-9_0",
"netbsd-386": "netbsd-386-9_0",
"netbsd-amd64": "netbsd-amd64-9_0",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"netbsd-arm": "netbsd-arm-bsiegert",
dashboard: add netbsd-arm64-bsiegert builder (reverse) Update golang/go#34036 Change-Id: Ic92093e764fa163d97a2f125a4534581adf217da Reviewed-on: https://go-review.googlesource.com/c/build/+/249137 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-08-18 21:42:30 +03:00
"netbsd-arm64": "netbsd-arm64-bsiegert",
dashboard: add nocgo SlowBot alias Fixes golang/go#42105. Change-Id: Ia36035e0e86a585d2a1ec5baac451c7219dbda79 Reviewed-on: https://go-review.googlesource.com/c/build/+/264634 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2020-10-23 23:18:30 +03:00
"nocgo": "linux-amd64-nocgo",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"openbsd": "openbsd-amd64-64",
"openbsd-386": "openbsd-386-64",
"openbsd-amd64": "openbsd-amd64-64",
dashboard: add openbsd-arm64-jsing builder (reverse) Update golang/go#31656 Fixes golang/go#34035 Change-Id: I7e3d20be148ad5111efeb78702352da14274938e Reviewed-on: https://go-review.googlesource.com/c/build/+/250117 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-20 21:42:01 +03:00
"openbsd-arm": "openbsd-arm-jsing",
"openbsd-arm64": "openbsd-arm64-jsing",
dashboard: add openbsd-mips64-jsing builder (reverse) Update golang/go#40995 Change-Id: Ie733438ab1823d65109e684a6a5e3098cdfdb252 Reviewed-on: https://go-review.googlesource.com/c/build/+/250118 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-23 19:02:40 +03:00
"openbsd-mips64": "openbsd-mips64-jsing",
dashboard: change "plan9" SlowBot alias to refer to plan9-arm The plan9-arm builder is currently passing fairly reliably on the dashboard, whereas the plan9-386-0intro builder is frequently failing with an "unexpected fault address". Until the plan9-386-0intro build is fixed, the plan9-arm builder seems more useful as a SlowBot to check architecture-agnostic changes that affect plan9. Updates golang/go#35456 Updates golang/go#22227 Change-Id: Ib5ecddafec91ca9274f9f0e15cb98621b41dda85 Reviewed-on: https://go-review.googlesource.com/c/build/+/233797 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-05-13 17:52:28 +03:00
"plan9": "plan9-arm",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"plan9-386": "plan9-386-0intro",
"plan9-amd64": "plan9-amd64-9front",
"ppc64": "linux-ppc64-buildlet",
"ppc64le": "linux-ppc64le-buildlet",
dashboard: add linux-riscv64-jsing builder (reverse) Also use this as the riscv64 slowbot, since linux-riscv64-unleashed has been MIA for quite some time. For golang/go#27532 Change-Id: I01309a66b96d4c35dbc9d1e08f55f5357b0acba0 Reviewed-on: https://go-review.googlesource.com/c/build/+/257917 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-09-27 21:25:41 +03:00
"riscv64": "linux-riscv64-jsing",
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
"s390x": "linux-s390x-ibm",
"solaris": "solaris-amd64-oraclerel",
"solaris-amd64": "solaris-amd64-oraclerel",
"wasm": "js-wasm",
"windows": "windows-amd64-2016",
"windows-386": "windows-386-2008",
"windows-amd64": "windows-amd64-2016",
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
// Builders are the different build configurations.
// The keys are like "darwin-amd64" or "linux-386-387".
// This map should not be modified by other packages.
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
// Initialization happens below, via calls to addBuilder.
all: change dashboard.BuildConfig to be a pointer type, like HostConfig BuildConfig grew way too large and has too many slices & such to be a value type. Make it a pointer type, which matches HostConfig. Change-Id: Ie625bece9d6d8c1ec6cff26e77416bd9b1f256d8 Reviewed-on: https://go-review.googlesource.com/c/145077 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 20:52:25 +03:00
var Builders = map[string]*BuildConfig{}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// Hosts contains the names and configs of all the types of
// buildlets. They can be VMs, containers, or dedicated machines.
var Hosts = map[string]*HostConfig{
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
"host-linux-jessie": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Debian Jessie, our standard Linux container image.",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
ContainerImage: "linux-x86-jessie:latest",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
"host-linux-stretch": &HostConfig{
Notes: "Debian Stretch",
ContainerImage: "linux-x86-stretch:latest",
dashboard: increase timeout scale for longtest builders, add more CPU Updates golang/go#25886 Change-Id: I5168e291ab77cbd3843bdc39e319a68dfa65aedd Reviewed-on: https://go-review.googlesource.com/c/build/+/167638 Reviewed-by: Russ Cox <rsc@golang.org>
2019-03-14 19:06:57 +03:00
machineType: "n1-standard-4", // 4 vCPUs, 15 GB mem
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
"host-linux-stretch-morecpu": &HostConfig{
dashboard: upsize machine type for windows-amd64-{longtest,race} builders Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for windows-amd64-longtest and windows-amd64-race builders. Continue to use the existing n1-highcpu-4 machine type for all other windows/386 and windows/amd64 builders for now. The primary motivation of this change is to increase amount of RAM available on the longtest builder from 3.6 GB to 7.2 GB, because it is consistenly failing due to being out of memory. Also fix minor typos, and increase consistency in comments and notes. Updates golang/go#33951 Change-Id: Iac5da67fa584059842850a9c8deda98c242741f2 Reviewed-on: https://go-review.googlesource.com/c/build/+/208500 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-11-23 02:01:25 +03:00
Notes: "Debian Stretch, but on n1-highcpu-16",
dashboard: increase timeout scale for longtest builders, add more CPU Updates golang/go#25886 Change-Id: I5168e291ab77cbd3843bdc39e319a68dfa65aedd Reviewed-on: https://go-review.googlesource.com/c/build/+/167638 Reviewed-by: Russ Cox <rsc@golang.org>
2019-03-14 19:06:57 +03:00
ContainerImage: "linux-x86-stretch:latest",
dashboard: change linux-amd64-longtest machine to n1-highcpu-16 CL 167638 made a change to add more CPU resources to the longtest builder, with the intention of going from 4 vCPUs, 15 GB RAM to 16 vCPUs, 14.4 GB RAM. It used n1-highcpu-8 GCE machine type, which actually has 8 vCPUs and 7.2 GB RAM.¹ Having less RAM than before wasn't the intention. Fix that by changing n1-highcpu-8 to n1-highcpu-16, which matches the comment. ¹ https://cloud.google.com/compute/docs/machine-types Updates golang/go#32831 Updates golang/go#33986 Updates golang/go#25886 Change-Id: I8426867fe33b3bf86576cb13d0d6113cd87f30c1 Reviewed-on: https://go-review.googlesource.com/c/build/+/192679 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-08-30 21:27:13 +03:00
machineType: "n1-highcpu-16", // 16 vCPUs, 14.4 GB mem
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "root",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
"host-linux-stretch-vmx": &HostConfig{
Notes: "Debian Stretch w/ Nested Virtualization (VMX CPU bit) enabled, for testing",
ContainerImage: "linux-x86-stretch:latest",
NestedVirt: true,
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
cmd/coordinator: cross-compile ARM build on Kubernetes Saves 4.5 minutes or so by using fast x86 machines to build the ARM build instead of running make.bash on Scaleway ARM machines. We still run the tests on ARM, and have a separate builder only running make.bash on ARM (see prior golang.org/cl/29670) Fixes golang/go#17105 Updates golang/go#17104 Change-Id: I1cb7b0e5b1cc8b644195f262328884ed3aff120a Reviewed-on: https://go-review.googlesource.com/29677 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-23 23:44:43 +03:00
"host-linux-armhf-cross": &HostConfig{
env/crosscompile: update the three cross compile envs from Jessie to Buster The earlier fix to remove IN_KUBERNETES in buildlet/stage0/stage0.go in CL 201739 caused Kubernetes-based cross compilation builds on fail to start up because I forgot to rebuild & push those container images. But when I went to re-push them, I found they're too old (Debian Jessie) and don't build anymore. So update them to Debian Buster. While I'm here, remove the Debian release name from their image name, as this won't be the last time we need to upgrade them. Unlike real builders, the OS release isn't the point of these containers, as they don't run tests. They exist only to do builds, so the OS isn't interesting or worth calling out. Updates golang/go#34956 Change-Id: Iee4691a821421b161a9333b4441f83ff45d1ffec Reviewed-on: https://go-review.googlesource.com/c/build/+/202021 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-10-19 07:02:04 +03:00
Notes: "Debian with armhf cross-compiler, built from env/crosscompile/linux-armhf",
ContainerImage: "linux-armhf-cross:latest",
cmd/coordinator: cross-compile ARM build on Kubernetes Saves 4.5 minutes or so by using fast x86 machines to build the ARM build instead of running make.bash on Scaleway ARM machines. We still run the tests on ARM, and have a separate builder only running make.bash on ARM (see prior golang.org/cl/29670) Fixes golang/go#17105 Updates golang/go#17104 Change-Id: I1cb7b0e5b1cc8b644195f262328884ed3aff120a Reviewed-on: https://go-review.googlesource.com/29677 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-23 23:44:43 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
cmd/coordinator, dashboard: cross compile linux-arm-arm5spacemonkey Change-Id: I3a228627ec8236d0d10bb9a1a61443c81493145d Reviewed-on: https://go-review.googlesource.com/38460 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-23 18:46:39 +03:00
"host-linux-armel-cross": &HostConfig{
env/crosscompile: update the three cross compile envs from Jessie to Buster The earlier fix to remove IN_KUBERNETES in buildlet/stage0/stage0.go in CL 201739 caused Kubernetes-based cross compilation builds on fail to start up because I forgot to rebuild & push those container images. But when I went to re-push them, I found they're too old (Debian Jessie) and don't build anymore. So update them to Debian Buster. While I'm here, remove the Debian release name from their image name, as this won't be the last time we need to upgrade them. Unlike real builders, the OS release isn't the point of these containers, as they don't run tests. They exist only to do builds, so the OS isn't interesting or worth calling out. Updates golang/go#34956 Change-Id: Iee4691a821421b161a9333b4441f83ff45d1ffec Reviewed-on: https://go-review.googlesource.com/c/build/+/202021 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-10-19 07:02:04 +03:00
Notes: "Debian with armel cross-compiler, from env/crosscompile/linux-armel",
ContainerImage: "linux-armel-cross:latest",
cmd/coordinator, dashboard: cross compile linux-arm-arm5spacemonkey Change-Id: I3a228627ec8236d0d10bb9a1a61443c81493145d Reviewed-on: https://go-review.googlesource.com/38460 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-23 18:46:39 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
"host-linux-amd64-localdev": &HostConfig{
IsReverse: true,
ExpectNum: 0,
Notes: "for localhost development of buildlets/gomote/coordinator only",
SSHUsername: os.Getenv("USER"),
},
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
"host-js-wasm": &HostConfig{
Notes: "Container with node.js for testing js/wasm.",
ContainerImage: "js-wasm:latest",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
dashboard, env, cmd/buildlet/testssh: fix gomote ssh for a number of buidlers And add a testssh tool to validate that SSH is working. Updates golang/go#32430 Change-Id: I5182419fa4db31b598f7a02412fb4ecc4060a796 Reviewed-on: https://go-review.googlesource.com/c/build/+/207459 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-15 23:31:23 +03:00
SSHUsername: "root",
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
},
dashboard: remove misleading -kube suffixes from host types They used to run on Kubernetes (and still could) but they're just a container and we run them on COS nowadays. Change-Id: I21a6db90efc5071997c6d6dcae1a42a7c28afa3a Reviewed-on: https://go-review.googlesource.com/c/build/+/209957 Reviewed-by: Alexander Rakoczy <alex@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-12-05 00:27:28 +03:00
"host-s390x-cross": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Container with s390x cross-compiler.",
env/crosscompile: update the three cross compile envs from Jessie to Buster The earlier fix to remove IN_KUBERNETES in buildlet/stage0/stage0.go in CL 201739 caused Kubernetes-based cross compilation builds on fail to start up because I forgot to rebuild & push those container images. But when I went to re-push them, I found they're too old (Debian Jessie) and don't build anymore. So update them to Debian Buster. While I'm here, remove the Debian release name from their image name, as this won't be the last time we need to upgrade them. Unlike real builders, the OS release isn't the point of these containers, as they don't run tests. They exist only to do builds, so the OS isn't interesting or worth calling out. Updates golang/go#34956 Change-Id: Iee4691a821421b161a9333b4441f83ff45d1ffec Reviewed-on: https://go-review.googlesource.com/c/build/+/202021 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-10-19 07:02:04 +03:00
ContainerImage: "linux-s390x-cross:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
},
env: add alpine linux builder Adds flag to upload to build binary statically. Updates golang/go#17891 Change-Id: If19b2e2cf9a84fa2b4cc6fdf55d8add5b54abff7 Reviewed-on: https://go-review.googlesource.com/33890 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-02 22:14:12 +03:00
"host-linux-x86-alpine": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Alpine container",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-alpine:latest",
dashboard, env, cmd/buildlet/testssh: fix gomote ssh for a number of buidlers And add a testssh tool to validate that SSH is working. Updates golang/go#32430 Change-Id: I5182419fa4db31b598f7a02412fb4ecc4060a796 Reviewed-on: https://go-review.googlesource.com/c/build/+/207459 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-15 23:31:23 +03:00
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.linux-amd64-static",
env: add alpine linux builder Adds flag to upload to build binary statically. Updates golang/go#17891 Change-Id: If19b2e2cf9a84fa2b4cc6fdf55d8add5b54abff7 Reviewed-on: https://go-review.googlesource.com/33890 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-02 22:14:12 +03:00
env: []string{"GOROOT_BOOTSTRAP=/usr/lib/go"},
dashboard, env, cmd/buildlet/testssh: fix gomote ssh for a number of buidlers And add a testssh tool to validate that SSH is working. Updates golang/go#32430 Change-Id: I5182419fa4db31b598f7a02412fb4ecc4060a796 Reviewed-on: https://go-review.googlesource.com/c/build/+/207459 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-15 23:31:23 +03:00
SSHUsername: "root",
env: add alpine linux builder Adds flag to upload to build binary statically. Updates golang/go#17891 Change-Id: If19b2e2cf9a84fa2b4cc6fdf55d8add5b54abff7 Reviewed-on: https://go-review.googlesource.com/33890 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-02 22:14:12 +03:00
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-linux-clang": &HostConfig{
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
Notes: "Container with clang.",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-clang:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
dashboard, env, cmd/buildlet/testssh: fix gomote ssh for a number of buidlers And add a testssh tool to validate that SSH is working. Updates golang/go#32430 Change-Id: I5182419fa4db31b598f7a02412fb4ecc4060a796 Reviewed-on: https://go-review.googlesource.com/c/build/+/207459 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-15 23:31:23 +03:00
SSHUsername: "root",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
"host-linux-sid": &HostConfig{
env/linux-x86-sid: convert from GCE VM to Kubernetes And update it. What we used to call "sid" was probably ancient. Change-Id: I76c1aad60eb655513119eaad0550ba5e0df3e9c5 Reviewed-on: https://go-review.googlesource.com/40399 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 04:54:09 +03:00
Notes: "Debian sid, updated occasionally.",
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
ContainerImage: "linux-x86-sid:latest",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
dashboard, env, cmd/buildlet/testssh: fix gomote ssh for a number of buidlers And add a testssh tool to validate that SSH is working. Updates golang/go#32430 Change-Id: I5182419fa4db31b598f7a02412fb4ecc4060a796 Reviewed-on: https://go-review.googlesource.com/c/build/+/207459 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-15 23:31:23 +03:00
SSHUsername: "root",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
env/linux-x86-fedora: add Fedora builder Not using a Fedora kernel yet, but better than nothing. Updates golang/go#26796 Change-Id: I228442bf6606696407a8d3fbd098df59df7353e1 Reviewed-on: https://go-review.googlesource.com/c/build/+/175999 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-08 19:25:38 +03:00
"host-linux-fedora": &HostConfig{
Notes: "Fedora 30",
ContainerImage: "linux-x86-fedora:latest",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/goboot"},
dashboard, env, cmd/buildlet/testssh: fix gomote ssh for a number of buidlers And add a testssh tool to validate that SSH is working. Updates golang/go#32430 Change-Id: I5182419fa4db31b598f7a02412fb4ecc4060a796 Reviewed-on: https://go-review.googlesource.com/c/build/+/207459 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-15 23:31:23 +03:00
SSHUsername: "root",
env/linux-x86-fedora: add Fedora builder Not using a Fedora kernel yet, but better than nothing. Updates golang/go#26796 Change-Id: I228442bf6606696407a8d3fbd098df59df7353e1 Reviewed-on: https://go-review.googlesource.com/c/build/+/175999 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-08 19:25:38 +03:00
},
dashboard: builders.go change for new scaleway docker config Working towards golang/go#20397 Change-Id: I0861f0a265498840cd7e852504cd5b9e53893036 Reviewed-on: https://go-review.googlesource.com/43813 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-22 23:17:48 +03:00
"host-linux-arm-scaleway": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
IsReverse: true,
HermeticReverse: true,
ExpectNum: 50,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
SSHUsername: "root",
dashboard: builders.go change for new scaleway docker config Working towards golang/go#20397 Change-Id: I0861f0a265498840cd7e852504cd5b9e53893036 Reviewed-on: https://go-review.googlesource.com/43813 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-22 23:17:48 +03:00
},
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
"host-linux-arm5spacemonkey": &HostConfig{
cmd/buildlet, cmd/coordinator: delete old --reverse mode All the buildlets have been updated to use --reverse-type with a host type instead of a builder type. Fixes golang/go#21260 Change-Id: I1264261f099c3686fe01455949486f523b94c6de Reviewed-on: https://go-review.googlesource.com/c/build/+/205608 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-06 23:34:49 +03:00
IsReverse: true,
ExpectNum: 3,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
OwnerGithub: "esnolte", // https://github.com/golang/go/issues/34973#issuecomment-543836871
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
},
dashboard: add linux-riscv64-jsing builder (reverse) Also use this as the riscv64 slowbot, since linux-riscv64-unleashed has been MIA for quite some time. For golang/go#27532 Change-Id: I01309a66b96d4c35dbc9d1e08f55f5357b0acba0 Reviewed-on: https://go-review.googlesource.com/c/build/+/257917 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-09-27 21:25:41 +03:00
"host-linux-riscv64-joelsing": &HostConfig{
Notes: "SiFive HiFive Unleashed RISC-V board. 8 GB RAM, 4 cores.",
IsReverse: true,
ExpectNum: 1,
OwnerGithub: "4a6f656c",
env: []string{"GOROOT_BOOTSTRAP=/usr/local/goboot"},
},
dashboard, cmd/coordinator: add linux-riscv64 builder, fix bug from CL 205608 Updates golang/go#27532 Updates golang/go#30262 Change-Id: If1593ab6e5653a9dce625f25a16bf8a082796592 Reviewed-on: https://go-review.googlesource.com/c/build/+/206639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-12 09:19:21 +03:00
"host-linux-riscv64-unleashed": &HostConfig{
Notes: "SiFive HiFive Unleashed RISC-V board. 8 GB RAM, 4 cores.",
IsReverse: true,
ExpectNum: 1, // for now. Joel's board might join the party later.
OwnerGithub: "bradfitz", // at home
env: []string{"GOROOT_BOOTSTRAP=/usr/local/goboot"},
},
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
"host-openbsd-amd64-62": &HostConfig{
VMImage: "openbsd-amd64-62",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64",
dashboard: set openbsd builders to use a Go 1.12 bootstrap toolchain Updates golang/go#30633 Change-Id: I593a66e18f7546fdeaf7513e2a16bdf82a5fa5a8 Reviewed-on: https://go-review.googlesource.com/c/build/+/165747 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-06 22:45:35 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-go1_12.tar.gz",
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
Notes: "OpenBSD 6.2; GCE VM is built from script in build/env/openbsd-amd64",
SSHUsername: "gopher",
},
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
"host-openbsd-386-62": &HostConfig{
VMImage: "openbsd-386-62-a",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386",
dashboard: set openbsd builders to use a Go 1.12 bootstrap toolchain Updates golang/go#30633 Change-Id: I593a66e18f7546fdeaf7513e2a16bdf82a5fa5a8 Reviewed-on: https://go-review.googlesource.com/c/build/+/165747 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-06 22:45:35 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-go1_12.tar.gz",
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
Notes: "OpenBSD 6.2; GCE VM is built from script in build/env/openbsd-386",
SSHUsername: "gopher",
},
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
"host-openbsd-amd64-64": &HostConfig{
env/openbsd-amd64: force tsc time source, double the build speed Fixes golang/go#29223 Change-Id: I43d1fc1eb6455c063d3842c330cb7d57295d7482 Reviewed-on: https://go-review.googlesource.com/c/160319 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-30 08:01:54 +03:00
VMImage: "openbsd-amd64-64-190129a",
buildlet, dashboard: add min CPU platform knob, require Skylake for OpenBSD I thought this would be enough for OpenBSD to select the TSC on its own without being forced to (as in CL 160319), but apparently it is not: https://github.com/golang/go/issues/29223#issuecomment-459035482 So it seems like we want both this CL and CL 160319. Updates golang/go#29223 Change-Id: I0a092d62881d8dcce0ef1129d8d32d8f4025b6ac Reviewed-on: https://go-review.googlesource.com/c/160457 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-30 20:13:07 +03:00
MinCPUPlatform: "Intel Skylake", // for better TSC? Maybe? see Issue 29223. builds faster at least.
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
machineType: "n1-highcpu-4",
cmd/buildlet: add, fix OpenBSD make targets Updates golang/go#35610 Change-Id: I736a439df10b455125688b6c26852c366f9283fe Reviewed-on: https://go-review.googlesource.com/c/build/+/207420 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-15 20:46:02 +03:00
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64",
dashboard: set openbsd builders to use a Go 1.12 bootstrap toolchain Updates golang/go#30633 Change-Id: I593a66e18f7546fdeaf7513e2a16bdf82a5fa5a8 Reviewed-on: https://go-review.googlesource.com/c/build/+/165747 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-06 22:45:35 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-go1_12.tar.gz",
env/openbsd-amd64: enable SMT on OpenBSD 6.4 Disable Spectre/Meltdown mitigations and enable simultaneous multithreading (SMT) on the OpenBSD 6.4 builder image. This was done by setting hw.smt=1 in /etc/sysctl.conf when building the image. I considered an alternative approach of doing this at runtime via a cmd/buildlet special case for OpenBSD. It would've likely been faster to implement if I had the idea at the beginning. However, by the time I saw it, I had already started to build a new OpenBSD 6.4 image with the /etc/sysctl.conf file, then tested it via debugnewvm. At this point, there's no longer a time saving advantage, so I decided to prefer the v2 image because it keeps OpenBSD-specific configuration more contained in env/openbsd-* directory, rather than spreading it out between there and the cmd/buildlet runtime code. It seems to be a slightly better option. The times to do a full build, as measured via cmd/debugnewvm, were: 6.2 = 19m25.218s 6.4 = 28m49.565s 6.4 with hw.smt=1 = 22m55.909s That should translate to faster trybot runs, which is important for open source project health. The 386 environment doesn't need to be updated because it doesn't support hw.smt: sysctl: hw.smt: value is not available Attempting to set it anyway should be harmless and okay to do, in case it happens to get supported in the future. We're only using amd64 environment for trybots, so it's okay for the purposes of golang/go#28403. Reference: https://man.openbsd.org/sysctl.2#HW_SMT_2 Fixes golang/go#28403 Change-Id: I7bc4cbf83ccbdb3aa9dd19eeabd88feb1c425811 Reviewed-on: https://go-review.googlesource.com/c/145022 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 20:05:59 +03:00
Notes: "OpenBSD 6.4 with hw.smt=1; GCE VM is built from script in build/env/openbsd-amd64",
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
SSHUsername: "gopher",
},
"host-openbsd-386-64": &HostConfig{
dashboard: update OpenBSD 6.4 builders to final release Previously, they were running a 6.4 snapshot release from October 4, 2018, added in CL 139857. The final release of OpenBSD 6.4 has been released on October 18, 2018, and can be used now. See https://www.openbsd.org/64.html. Fixup unbound variable in env/openbsd-amd64/make.bash: $ ./make.bash ./make.bash: line 57: PKG_ADD_OPTIONS: unbound variable The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: I0e3b1ed5aa9b2e624d9afbe46a8a344f88ddcc56 Reviewed-on: https://go-review.googlesource.com/c/143458 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-19 22:50:28 +03:00
VMImage: "openbsd-386-64",
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
machineType: "n1-highcpu-4",
cmd/buildlet: add, fix OpenBSD make targets Updates golang/go#35610 Change-Id: I736a439df10b455125688b6c26852c366f9283fe Reviewed-on: https://go-review.googlesource.com/c/build/+/207420 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-15 20:46:02 +03:00
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386",
dashboard: set openbsd builders to use a Go 1.12 bootstrap toolchain Updates golang/go#30633 Change-Id: I593a66e18f7546fdeaf7513e2a16bdf82a5fa5a8 Reviewed-on: https://go-review.googlesource.com/c/build/+/165747 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-06 22:45:35 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-go1_12.tar.gz",
dashboard: update OpenBSD 6.4 builders to final release Previously, they were running a 6.4 snapshot release from October 4, 2018, added in CL 139857. The final release of OpenBSD 6.4 has been released on October 18, 2018, and can be used now. See https://www.openbsd.org/64.html. Fixup unbound variable in env/openbsd-amd64/make.bash: $ ./make.bash ./make.bash: line 57: PKG_ADD_OPTIONS: unbound variable The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: I0e3b1ed5aa9b2e624d9afbe46a8a344f88ddcc56 Reviewed-on: https://go-review.googlesource.com/c/143458 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-19 22:50:28 +03:00
Notes: "OpenBSD 6.4; GCE VM is built from script in build/env/openbsd-386",
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
SSHUsername: "gopher",
},
dashboard: add openbsd-arm-jsing builder This will allow the openbsd-arm builder to be migrated to a buildlet. Change-Id: If38788daf2d1e7f237ebdcce384a098e08bffb0b Reviewed-on: https://go-review.googlesource.com/c/build/+/192598 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-30 20:09:05 +03:00
"host-openbsd-arm-joelsing": &HostConfig{
IsReverse: true,
ExpectNum: 1,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
OwnerGithub: "4a6f656c",
},
dashboard: add openbsd-arm64-jsing builder (reverse) Update golang/go#31656 Fixes golang/go#34035 Change-Id: I7e3d20be148ad5111efeb78702352da14274938e Reviewed-on: https://go-review.googlesource.com/c/build/+/250117 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-20 21:42:01 +03:00
"host-openbsd-arm64-joelsing": &HostConfig{
IsReverse: true,
ExpectNum: 1,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
OwnerGithub: "4a6f656c",
},
dashboard: add openbsd-mips64-jsing builder (reverse) Update golang/go#40995 Change-Id: Ie733438ab1823d65109e684a6a5e3098cdfdb252 Reviewed-on: https://go-review.googlesource.com/c/build/+/250118 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-23 19:02:40 +03:00
"host-openbsd-mips64-joelsing": &HostConfig{
IsReverse: true,
ExpectNum: 1,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
OwnerGithub: "4a6f656c",
},
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
"host-freebsd-11_1": &HostConfig{
dashboard: rebuild FreeBSD VM to avoid 10 second boot prompt FreeBSD VMs boot up 10 seconds faster now, and shut down faster. Updates golang/go#22894 Updates golang/go#22895 Change-Id: I8eef979fc43d95d3c03104a761e70ec5737a3a97 Reviewed-on: https://go-review.googlesource.com/80304 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-29 01:07:29 +03:00
VMImage: "freebsd-amd64-111-b",
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1. freebsd.org says that the current supported releases are: > Production: 11.1, 11.0, 10.4, 10.3 Previously we had builders for 9.3, 10.1, and 11.0. We'll keep FreeBSD 9.3 for now (as it's still supported by the last two released Go versions), but we'll stop testing FreeBSD 9.3 at tip shortly here (that will be a separate change). This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3, spanning the range from the old to newest supported release. We'll stop testing 10.4 and 11.0, as that's a bit overkill. Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9 release notes (https://golang.org/doc/go1.9#freebsd) said that future Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test FreeBSD 10.3, being the oldest. (Go has historically "supported" FreeBSD releases a bit longer than they're officially supported by FreeBSD, partly due to our out-of-sync release cycles.) Fixes golang/go#19303 Fixes golang/go#22854 Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f Reviewed-on: https://go-review.googlesource.com/80297 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
Notes: "FreeBSD 11.1; GCE VM is built from script in build/env/freebsd-amd64",
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
machineType: "n1-highcpu-4",
dashboard: upsize freebsd-amd64-race builder to 7.2 GB RAM Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for the freebsd-amd64-race builder. The freebsd-amd64-race builder has produced good test results for the x/tools repo for a long time, but by now it has started to consistently fail for reasons that seem connected to it having only 3.6 GB memory. The Windows race builders needed to be bumped from 7.2 GB to 14.4 GB to run successfully, so this change makes a small incremental step to bring freebsd-amd64-race closer in line with other builders. If memory-related problems continue to occur with 7.2 GB, the next step will be to go up to 14.4 GB. The freebsd-amd64-race builder is using an older version of FreeBSD. We may want to start using a newer one for running tests with -race, but that should be a separate change so we can see the results of this change without another confounding variable. Also update all FreeBSD builders to use https in buildletURLTmpl, because it's expected to work fine and will be more consistent. Updates golang/go#36444 Updates golang/go#34621 Updates golang/go#29252 Updates golang/go#33986 Change-Id: Idfcefd1c91bddc9f70ab23e02fcdca54fda9d1ac Reviewed-on: https://go-review.googlesource.com/c/build/+/214433 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-01-13 19:26:43 +03:00
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
env: []string{"CC=clang"},
SSHUsername: "gopher",
},
"host-freebsd-11_1-big": &HostConfig{
VMImage: "freebsd-amd64-111-b",
dashboard: upsize freebsd-amd64-race builder to 16 vCPUs, 14.4 GB RAM This is a followup to CL 214433. Start using n1-highcpu-16 machine type instead of n1-highcpu-8 for the freebsd-amd64-race builder. Increasing the RAM from 3.6 GB to 7.2 GB has helped golang/go#36444 significantly: the builder stopped failing consistently on x/tools and resulted in many data races being uncovered in golang/go#36605. However, by now, it has started to fail consistently again. This time it seems to be due to low performance, causing the tests in golang.org/x/tools/internal/lsp/regtest package to fail due with "context deadline exceeded" errors. FreeBSD is one of the ports that stays visible when "show only first- class ports" is checked on build.golang.org. The other -race builders have all been upgraded to the n1-highcpu-16 machine type by now out of necessity. It seems fair to provide the FreeBSD port with an equal amount of resources, even if the increased memory isn't strictly required yet. Once this change is applied, if the failures persist, we can be more confident that the problem is due to the code or the port, rather than due to this -race builder having 2𝗑 less CPU and RAM resources compared to other -race builders. An alternative is to increase timeout for this builder type, but I'm opting to defer exploring that after equalizing the machine type. For golang/go#36444. For golang/go#34621. For golang/go#29252. For golang/go#33986. Change-Id: I41f149365128c7bc6f576c778ac07618acc04612 Reviewed-on: https://go-review.googlesource.com/c/build/+/227859 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-10 18:47:55 +03:00
Notes: "Same as host-freebsd-11_1, but on n1-highcpu-16",
machineType: "n1-highcpu-16", // 16 vCPUs, 14.4 GB mem
dashboard: upsize freebsd-amd64-race builder to 7.2 GB RAM Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for the freebsd-amd64-race builder. The freebsd-amd64-race builder has produced good test results for the x/tools repo for a long time, but by now it has started to consistently fail for reasons that seem connected to it having only 3.6 GB memory. The Windows race builders needed to be bumped from 7.2 GB to 14.4 GB to run successfully, so this change makes a small incremental step to bring freebsd-amd64-race closer in line with other builders. If memory-related problems continue to occur with 7.2 GB, the next step will be to go up to 14.4 GB. The freebsd-amd64-race builder is using an older version of FreeBSD. We may want to start using a newer one for running tests with -race, but that should be a separate change so we can see the results of this change without another confounding variable. Also update all FreeBSD builders to use https in buildletURLTmpl, because it's expected to work fine and will be more consistent. Updates golang/go#36444 Updates golang/go#34621 Updates golang/go#29252 Updates golang/go#33986 Change-Id: Idfcefd1c91bddc9f70ab23e02fcdca54fda9d1ac Reviewed-on: https://go-review.googlesource.com/c/build/+/214433 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-01-13 19:26:43 +03:00
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
env: []string{"CC=clang"},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
},
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
"host-freebsd-11_2": &HostConfig{
VMImage: "freebsd-amd64-112",
Notes: "FreeBSD 11.2; GCE VM is built from script in build/env/freebsd-amd64",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
SSHUsername: "gopher",
},
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
"host-freebsd-12_0": &HostConfig{
env/freebsd-amd64: update FreeBSD 12.0 to a -RELEASE image Install GDB from the ports collection, the in-tree was removed from the base system in FreeBSD 12.0, and the older releases have an unsupported (6.1.1 or older version). Updates golang/go#25289 Change-Id: I8899a2a70c0b9ee9171621206431ca24a7e97704 Reviewed-on: https://go-review.googlesource.com/c/155977 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-12-29 20:53:25 +03:00
VMImage: "freebsd-amd64-120-v1",
Notes: "FreeBSD 12.0; GCE VM is built from script in build/env/freebsd-amd64",
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
SSHUsername: "gopher",
},
dashboard, env/netbsd-{amd64,386}: add NetBSD 9.0 builders Update NetBSD amd64 and 386 builders to the latest NetBSD 9.0 release, using the latest versions of anita (2.3) and pkgsrc packages (2019Q4). Also increase the image disk size from 4 GB to 16 GB. The motivation is to help tests that require more disk space and large dependencies. Additional adjustments to the image building script will need to be made to make use of increased disk space for the /tmp volume, as it is currently being mounted in RAM with size equal to 25% of memory. Remove the manual invocation of the mozilla-rootcerts install command. It no longer works as is in NetBSD 9.0. The mozilla-rootcerts-openssl package is added instead, which successfully performs that same task. Hopefully this will be a more robust approach for the future. Continue to use an explicit URL for the NetBSD release download, and remove unused find_latest_release. The plan is to roll out these builders first (without trybots), gain more confidence, enable trybots, and then remove NetBSD 8.0 builders. For golang/go#37244. For golang/go#30839. For golang/go#31858. For golang/go#31726. For golang/go#38283. Change-Id: Id41278eaebe64cfbcf7b1ab9bfd64e550ee0bbf1 Reviewed-on: https://go-review.googlesource.com/c/build/+/226977 Reviewed-by: Benny Siegert <bsiegert@gmail.com> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-02 15:55:30 +03:00
"host-netbsd-amd64-9_0": &HostConfig{
VMImage: "netbsd-amd64-9-0-2019q4",
Notes: "NetBSD 9.0; GCE VM is built from script in build/env/netbsd-amd64",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.netbsd-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-netbsd-amd64-2da6b33.tar.gz",
SSHUsername: "root",
},
"host-netbsd-386-9_0": &HostConfig{
VMImage: "netbsd-i386-9-0-2019q4",
Notes: "NetBSD 9.0; GCE VM is built from script in build/env/netbsd-386",
machineType: "n1-highcpu-4",
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.netbsd-386",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-netbsd-386-0b3b511.tar.gz",
SSHUsername: "root",
},
dashboard: add netbsd-arm-bsiegert This seems to be the last builder still using the old builder binary and not buildlet. It needs an entry in here to switch to buildlet and stage0. Change-Id: If2fdd2298ab88d56b6850061050caf27ca7c8983 Reviewed-on: https://go-review.googlesource.com/c/build/+/168061 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-19 16:28:04 +03:00
"host-netbsd-arm-bsiegert": &HostConfig{
IsReverse: true,
ExpectNum: 1,
env: []string{"GOROOT_BOOTSTRAP=/usr/pkg/go112"},
OwnerGithub: "bsiegert",
},
dashboard: add netbsd-arm64-bsiegert builder (reverse) Update golang/go#34036 Change-Id: Ic92093e764fa163d97a2f125a4534581adf217da Reviewed-on: https://go-review.googlesource.com/c/build/+/249137 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-08-18 21:42:30 +03:00
"host-netbsd-arm64-bsiegert": &HostConfig{
IsReverse: true,
ExpectNum: 1,
env: []string{"GOROOT_BOOTSTRAP=/usr/pkg/go114"},
OwnerGithub: "bsiegert",
},
dashboard: rename builder dragonfly-amd64-5_6 to dragonfly-amd64-5_8 * DragonFly BSD 5.8 was released on March 3rd, 2020. This is the next release after 5.6. * Gobuilder VM has been upgraded to 5.8 and the scripts adjusted. Change-Id: I1d885359aedef8182e57f1ea004982f5060ae7af GitHub-Last-Rev: 25fa394d4c1cfa4a0198e8be0991dcae5133b90e GitHub-Pull-Request: golang/build#27 Reviewed-on: https://go-review.googlesource.com/c/build/+/222517 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-04-04 13:40:16 +03:00
"host-dragonfly-amd64-5_8": &HostConfig{
dashboard: update Dragonfly tip policy for ABI change, add release builder From https://github.com/golang/go/issues/34958#issuecomment-543852995 : > Go's DragonFly support policy is that we support the latest stable > release primarily, but also try to keep DragonFly master passing, in > prep for it to become the latest stable release. > > But that does mean we need one more builder at the moment. Updates golang/go#34958 Updates golang/go#23060 Change-Id: I84be7c64eac593dee2252c397f9529deea13605a Reviewed-on: https://go-review.googlesource.com/c/build/+/202478 Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 18:00:32 +03:00
IsReverse: true,
ExpectNum: 1,
dashboard: transfer ownership of DragonflyBSD 5.6 builder Change-Id: I9077f043ba43192fad2b2151ae37ab41c1d27aa9 Reviewed-on: https://go-review.googlesource.com/c/build/+/208717 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-25 16:28:27 +03:00
Notes: "DragonFly BSD release version, run by DragonFly team",
dashboard: update Dragonfly tip policy for ABI change, add release builder From https://github.com/golang/go/issues/34958#issuecomment-543852995 : > Go's DragonFly support policy is that we support the latest stable > release primarily, but also try to keep DragonFly master passing, in > prep for it to become the latest stable release. > > But that does mean we need one more builder at the moment. Updates golang/go#34958 Updates golang/go#23060 Change-Id: I84be7c64eac593dee2252c397f9529deea13605a Reviewed-on: https://go-review.googlesource.com/c/build/+/202478 Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 18:00:32 +03:00
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
dashboard: enable SSH access in DragonFly BSD builders Fixes golang/go#38796. Change-Id: If84c7ba39fe96ce0b93d818ae9f163338d954976 GitHub-Last-Rev: 3548bbc15722b14d819c46f5dabef0ae4a7455b7 GitHub-Pull-Request: golang/build#29 Reviewed-on: https://go-review.googlesource.com/c/build/+/232337 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-05-05 22:18:12 +03:00
SSHUsername: "root",
dashboard: transfer ownership of DragonflyBSD 5.6 builder Change-Id: I9077f043ba43192fad2b2151ae37ab41c1d27aa9 Reviewed-on: https://go-review.googlesource.com/c/build/+/208717 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-25 16:28:27 +03:00
OwnerGithub: "tuxillo",
dashboard: update Dragonfly tip policy for ABI change, add release builder From https://github.com/golang/go/issues/34958#issuecomment-543852995 : > Go's DragonFly support policy is that we support the latest stable > release primarily, but also try to keep DragonFly master passing, in > prep for it to become the latest stable release. > > But that does mean we need one more builder at the moment. Updates golang/go#34958 Updates golang/go#23060 Change-Id: I84be7c64eac593dee2252c397f9529deea13605a Reviewed-on: https://go-review.googlesource.com/c/build/+/202478 Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 18:00:32 +03:00
},
dashboard: rename DragonFlyBSD host type, change owners Change-Id: Iff3de20c4bb6a017f204602542652f8533b599e7 Reviewed-on: https://go-review.googlesource.com/c/build/+/205460 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-05 21:55:17 +03:00
"host-dragonfly-amd64-master": &HostConfig{
IsReverse: true,
ExpectNum: 1,
Notes: "DragonFly BSD master, run by DragonFly team",
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
dashboard: enable SSH access in DragonFly BSD builders Fixes golang/go#38796. Change-Id: If84c7ba39fe96ce0b93d818ae9f163338d954976 GitHub-Last-Rev: 3548bbc15722b14d819c46f5dabef0ae4a7455b7 GitHub-Pull-Request: golang/build#29 Reviewed-on: https://go-review.googlesource.com/c/build/+/232337 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-05-05 22:18:12 +03:00
SSHUsername: "root",
dashboard: rename DragonFlyBSD host type, change owners Change-Id: Iff3de20c4bb6a017f204602542652f8533b599e7 Reviewed-on: https://go-review.googlesource.com/c/build/+/205460 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-05 21:55:17 +03:00
OwnerGithub: "tuxillo",
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
},
"host-freebsd-arm-paulzhol": &HostConfig{
cmd/buildlet, cmd/coordinator: delete old --reverse mode All the buildlets have been updated to use --reverse-type with a host type instead of a builder type. Fixes golang/go#21260 Change-Id: I1264261f099c3686fe01455949486f523b94c6de Reviewed-on: https://go-review.googlesource.com/c/build/+/205608 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-06 23:34:49 +03:00
IsReverse: true,
ExpectNum: 1,
Notes: "Cubiboard2 1Gb RAM dual-core Cortex-A7 (Allwinner A20), FreeBSD 11.1-RELEASE",
env: []string{"GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4"},
OwnerGithub: "paulzhol",
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
},
dashboard: add FreeBSD arm64 builder Updates golang/go#24715 Change-Id: I8eb3f6372d69718b6f453e5415cdf16aa00ccae6 Reviewed-on: https://go-review.googlesource.com/c/build/+/198597 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-10-03 15:56:19 +03:00
"host-freebsd-arm64-dmgk": &HostConfig{
IsReverse: true,
ExpectNum: 1,
Notes: "AWS EC2 a1.large 2 vCPU 4GiB RAM, FreeBSD 12.1-STABLE",
env: []string{"GOROOT_BOOTSTRAP=/usr/home/builder/gobootstrap"},
OwnerGithub: "dmgk",
},
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
"host-plan9-arm-0intro": &HostConfig{
IsReverse: true,
ExpectNum: 1,
Notes: "Raspberry Pi 3 Model B, Plan 9 from Bell Labs",
OwnerGithub: "0intro",
},
"host-plan9-amd64-0intro": &HostConfig{
IsReverse: true,
ExpectNum: 1,
OwnerGithub: "0intro",
},
dashboard: add plan9-386-0intro builder This change adds the plan9-386-0intro builder configuration. Updates golang/go#29801. Change-Id: I5f64f4592a499eb1691498805fd0dfa5082c33d5 Reviewed-on: https://go-review.googlesource.com/c/build/+/175901 Run-TryBot: David du Colombier <0intro@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-09 19:52:40 +03:00
"host-plan9-386-0intro": &HostConfig{
IsReverse: true,
ExpectNum: 1,
OwnerGithub: "0intro",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-plan9-386-gce": &HostConfig{
env/plan9: revert SMP changes, stop running cmd/go tests Revert "env: enable SMP in Plan 9 image" This reverts commit 6478d3e7bdff4c8d810ba44f970ca4b0bfc92405. Revert "dashboard: bump plan9 GCE image" This reverts commit ce208996bf954a87d68789bb6a9000256f016c77. Updates golang/go#29822 Change-Id: I47e75d135e092ee9d5a80c66e91c1048c67cd5f6 Reviewed-on: https://go-review.googlesource.com/c/158542 Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: David du Colombier <0intro@gmail.com>
2019-01-19 01:19:58 +03:00
VMImage: "plan9-386-v7",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "Plan 9 from 0intro; GCE VM is built from script in build/env/plan9-386",
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.plan9-386",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-plan9-386.tar.gz",
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// We *were* using n1-standard-1 because Plan 9 can only
// reliably use a single CPU. Using 2 or 4 and we see
// test failures. See:
// https://golang.org/issue/8393
// https://golang.org/issue/9491
// n1-standard-1 has 3.6 GB of memory which WAS (see below)
// overkill (userspace probably only sees 2GB anyway),
// but it's the cheapest option. And plenty to keep
// our ~250 MB of inputs+outputs in its ramfs.
//
// But the docs says "For the n1 series of machine
// types, a virtual CPU is implemented as a single
// hyperthread on a 2.6GHz Intel Sandy Bridge Xeon or
// Intel Ivy Bridge Xeon (or newer) processor. This
// means that the n1-standard-2 machine type will see
// a whole physical core."
//
// ... so we used n1-highcpu-2 (1.80 RAM, still
// plenty), just so we can get 1 whole core for the
// single-core Plan 9. It will see 2 virtual cores and
// only use 1, but we hope that 1 will be more powerful
// and we'll stop timing out on tests.
machineType: "n1-highcpu-4",
dashboard: increase plan9's GO_TEST_TIMEOUT_SCALE Updates golang/go#29801 Change-Id: Ib346d717ae9b8de18666d6ea543d9bdcd2856f3b Reviewed-on: https://go-review.googlesource.com/c/158537 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-01-18 19:14:12 +03:00
env: []string{"GO_TEST_TIMEOUT_SCALE=3"},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
"host-windows-amd64-2008": &HostConfig{
env/windows: disable windows defender on builders Windows Defender scans all new files written to disk which slows down build performance. - Uninstall Windows Defender as part of image building - Refactor/Cleanup registry section for consistency - Promote new image version to dashboard (built in staging/prod) fixes golang/go#26055 Change-Id: I3e16b9a9581824c20abed5e8ffae1efd46c6dd09 Reviewed-on: https://go-review.googlesource.com/121937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-02 20:49:55 +03:00
VMImage: "windows-amd64-server-2008r2-v7",
dashboard: upsize machine type for windows-amd64-{longtest,race} builders Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for windows-amd64-longtest and windows-amd64-race builders. Continue to use the existing n1-highcpu-4 machine type for all other windows/386 and windows/amd64 builders for now. The primary motivation of this change is to increase amount of RAM available on the longtest builder from 3.6 GB to 7.2 GB, because it is consistenly failing due to being out of memory. Also fix minor typos, and increase consistency in comments and notes. Updates golang/go#33951 Change-Id: Iac5da67fa584059842850a9c8deda98c242741f2 Reviewed-on: https://go-review.googlesource.com/c/build/+/208500 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-11-23 02:01:25 +03:00
machineType: "n1-highcpu-4", // 4 vCPUs, 3.6 GB mem
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
SSHUsername: "gopher",
},
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
"host-windows-amd64-2012": &HostConfig{
env/windows: disable windows defender on builders Windows Defender scans all new files written to disk which slows down build performance. - Uninstall Windows Defender as part of image building - Refactor/Cleanup registry section for consistency - Promote new image version to dashboard (built in staging/prod) fixes golang/go#26055 Change-Id: I3e16b9a9581824c20abed5e8ffae1efd46c6dd09 Reviewed-on: https://go-review.googlesource.com/121937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-02 20:49:55 +03:00
VMImage: "windows-amd64-server-2012r2-v7",
dashboard: upsize machine type for windows-amd64-{longtest,race} builders Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for windows-amd64-longtest and windows-amd64-race builders. Continue to use the existing n1-highcpu-4 machine type for all other windows/386 and windows/amd64 builders for now. The primary motivation of this change is to increase amount of RAM available on the longtest builder from 3.6 GB to 7.2 GB, because it is consistenly failing due to being out of memory. Also fix minor typos, and increase consistency in comments and notes. Updates golang/go#33951 Change-Id: Iac5da67fa584059842850a9c8deda98c242741f2 Reviewed-on: https://go-review.googlesource.com/c/build/+/208500 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-11-23 02:01:25 +03:00
machineType: "n1-highcpu-4", // 4 vCPUs, 3.6 GB mem
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
dashboard,cmd/buildlet,cmd/coordinator: public key auth on windows Enable public key auth for windows buildlets. Previously the username/password was hardcoded. This change enables the `/connect-ssh` http endpoint on the buildlet to setup authorized_keys and uses the coordinator to proxy ssh traffic to the buildlet. Validation: - run local coordinator (with patches to dev env to act like GCE) with changes - build windows buildlet, post to own bucket - execute: gomote -localdev create windows-amd64-2012 ssh -p 2222 user-jrjohnson-windows-amd64-2012-0@127.0.0.1 Fixes golang/go#21247 Change-Id: If7adc7e6296a071d50412d9fa9d46ecd9e6cc3f3 Reviewed-on: https://go-review.googlesource.com/52350 Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-08-01 02:55:28 +03:00
SSHUsername: "gopher",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
},
"host-windows-amd64-2016": &HostConfig{
env/windows: disable windows defender on builders Windows Defender scans all new files written to disk which slows down build performance. - Uninstall Windows Defender as part of image building - Refactor/Cleanup registry section for consistency - Promote new image version to dashboard (built in staging/prod) fixes golang/go#26055 Change-Id: I3e16b9a9581824c20abed5e8ffae1efd46c6dd09 Reviewed-on: https://go-review.googlesource.com/121937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-02 20:49:55 +03:00
VMImage: "windows-amd64-server-2016-v7",
dashboard: upsize machine type for windows-amd64-{longtest,race} builders Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for windows-amd64-longtest and windows-amd64-race builders. Continue to use the existing n1-highcpu-4 machine type for all other windows/386 and windows/amd64 builders for now. The primary motivation of this change is to increase amount of RAM available on the longtest builder from 3.6 GB to 7.2 GB, because it is consistenly failing due to being out of memory. Also fix minor typos, and increase consistency in comments and notes. Updates golang/go#33951 Change-Id: Iac5da67fa584059842850a9c8deda98c242741f2 Reviewed-on: https://go-review.googlesource.com/c/build/+/208500 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-11-23 02:01:25 +03:00
machineType: "n1-highcpu-4", // 4 vCPUs, 3.6 GB mem
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
SSHUsername: "gopher",
},
"host-windows-amd64-2016-big": &HostConfig{
dashboard: upsize windows-amd64-{longtest,race} builders to 14.4 GB RAM Start using n1-highcpu-16 machine type instead of n1-highcpu-8 for windows-amd64-longtest and windows-amd64-race builders. The primary motivation of this change is to increase amount of RAM available on the race builder from 7.2 GB to 14.4 GB. CL 208500 has helped by going from 3.6 to 7.2, but empirically 8 GB is not enough for all tests to pass, while 16 GB can be. We already use 16 GB for some other race and/or long builders. Updates golang/go#35186 Updates golang/go#33951 Updates golang/go#33986 Change-Id: I8031a517e6e44ce5e5faba085e7c781a1fcab92e Reviewed-on: https://go-review.googlesource.com/c/build/+/213557 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2020-01-07 07:55:13 +03:00
Notes: "Same as host-windows-amd64-2016, but on n1-highcpu-16",
dashboard: upsize machine type for windows-amd64-{longtest,race} builders Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for windows-amd64-longtest and windows-amd64-race builders. Continue to use the existing n1-highcpu-4 machine type for all other windows/386 and windows/amd64 builders for now. The primary motivation of this change is to increase amount of RAM available on the longtest builder from 3.6 GB to 7.2 GB, because it is consistenly failing due to being out of memory. Also fix minor typos, and increase consistency in comments and notes. Updates golang/go#33951 Change-Id: Iac5da67fa584059842850a9c8deda98c242741f2 Reviewed-on: https://go-review.googlesource.com/c/build/+/208500 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-11-23 02:01:25 +03:00
VMImage: "windows-amd64-server-2016-v7",
dashboard: upsize windows-amd64-{longtest,race} builders to 14.4 GB RAM Start using n1-highcpu-16 machine type instead of n1-highcpu-8 for windows-amd64-longtest and windows-amd64-race builders. The primary motivation of this change is to increase amount of RAM available on the race builder from 7.2 GB to 14.4 GB. CL 208500 has helped by going from 3.6 to 7.2, but empirically 8 GB is not enough for all tests to pass, while 16 GB can be. We already use 16 GB for some other race and/or long builders. Updates golang/go#35186 Updates golang/go#33951 Updates golang/go#33986 Change-Id: I8031a517e6e44ce5e5faba085e7c781a1fcab92e Reviewed-on: https://go-review.googlesource.com/c/build/+/213557 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2020-01-07 07:55:13 +03:00
machineType: "n1-highcpu-16", // 16 vCPUs, 14.4 GB mem
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
dashboard,cmd/buildlet,cmd/coordinator: public key auth on windows Enable public key auth for windows buildlets. Previously the username/password was hardcoded. This change enables the `/connect-ssh` http endpoint on the buildlet to setup authorized_keys and uses the coordinator to proxy ssh traffic to the buildlet. Validation: - run local coordinator (with patches to dev env to act like GCE) with changes - build windows buildlet, post to own bucket - execute: gomote -localdev create windows-amd64-2012 ssh -p 2222 user-jrjohnson-windows-amd64-2012-0@127.0.0.1 Fixes golang/go#21247 Change-Id: If7adc7e6296a071d50412d9fa9d46ecd9e6cc3f3 Reviewed-on: https://go-review.googlesource.com/52350 Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-08-01 02:55:28 +03:00
SSHUsername: "gopher",
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-darwin-10_11": &HostConfig{
IsReverse: true,
dashboard, cmd/makemac: add macOS 10.15 Catalina builder Added the macOS 10.15 Catalina builder to the dashboard. Catalina builders should use the Mojave guestid in vSphere 6.7 update 3 because the Catalina guestid is not available in this version of vSphere. A Catalina builder will run on this version of vSphere with the Mojave guestid. Modified the expected number of vm instances running on Macstadium due to the addition of macOS 10.15 builders. Updates golang/go#34748 Change-Id: If14f6e088afc05dd1783460f21907e9345e3c338 Reviewed-on: https://go-review.googlesource.com/c/build/+/207937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-11-19 17:55:01 +03:00
ExpectNum: 3,
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "MacStadium OS X 10.11 VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
HermeticReverse: false, // TODO: make it so, like 10.12
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
"host-darwin-10_12": &HostConfig{
IsReverse: true,
dashboard: remove OS X 10.10 builders OS X 10.10 is no longer supported by the latest versions of Go (1.13 and 1.14). It is no longer in use. Fixes golang/go#37713 Change-Id: Ibc4a527c17f2dafd00ee0fba2e312394e1ceac9c Reviewed-on: https://go-review.googlesource.com/c/build/+/222337 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-03-06 20:14:44 +03:00
ExpectNum: 4,
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
Notes: "MacStadium OS X 10.12 VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
},
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername: "gopher",
HermeticReverse: true, // we destroy the VM when done & let cmd/makemac recreate
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
},
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
"host-darwin-10_14": &HostConfig{
IsReverse: true,
dashboard: remove OS X 10.10 builders OS X 10.10 is no longer supported by the latest versions of Go (1.13 and 1.14). It is no longer in use. Fixes golang/go#37713 Change-Id: Ibc4a527c17f2dafd00ee0fba2e312394e1ceac9c Reviewed-on: https://go-review.googlesource.com/c/build/+/222337 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-03-06 20:14:44 +03:00
ExpectNum: 6,
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
Notes: "MacStadium macOS Mojave (10.14) VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/goboot", // Go 1.12.1
},
SSHUsername: "gopher",
HermeticReverse: true, // we destroy the VM when done & let cmd/makemac recreate
},
dashboard, cmd/makemac: add macOS 10.15 Catalina builder Added the macOS 10.15 Catalina builder to the dashboard. Catalina builders should use the Mojave guestid in vSphere 6.7 update 3 because the Catalina guestid is not available in this version of vSphere. A Catalina builder will run on this version of vSphere with the Mojave guestid. Modified the expected number of vm instances running on Macstadium due to the addition of macOS 10.15 builders. Updates golang/go#34748 Change-Id: If14f6e088afc05dd1783460f21907e9345e3c338 Reviewed-on: https://go-review.googlesource.com/c/build/+/207937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-11-19 17:55:01 +03:00
"host-darwin-10_15": &HostConfig{
IsReverse: true,
dashboard: remove OS X 10.10 builders OS X 10.10 is no longer supported by the latest versions of Go (1.13 and 1.14). It is no longer in use. Fixes golang/go#37713 Change-Id: Ibc4a527c17f2dafd00ee0fba2e312394e1ceac9c Reviewed-on: https://go-review.googlesource.com/c/build/+/222337 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-03-06 20:14:44 +03:00
ExpectNum: 7,
dashboard, cmd/makemac: add macOS 10.15 Catalina builder Added the macOS 10.15 Catalina builder to the dashboard. Catalina builders should use the Mojave guestid in vSphere 6.7 update 3 because the Catalina guestid is not available in this version of vSphere. A Catalina builder will run on this version of vSphere with the Mojave guestid. Modified the expected number of vm instances running on Macstadium due to the addition of macOS 10.15 builders. Updates golang/go#34748 Change-Id: If14f6e088afc05dd1783460f21907e9345e3c338 Reviewed-on: https://go-review.googlesource.com/c/build/+/207937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-11-19 17:55:01 +03:00
Notes: "MacStadium macOS Catalina (10.15) VM under VMWare ESXi",
env: []string{
"GOROOT_BOOTSTRAP=/Users/gopher/goboot", // Go 1.12.1
},
SSHUsername: "gopher",
HermeticReverse: true, // we destroy the VM when done & let cmd/makemac recreate
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-linux-s390x": &HostConfig{
cmd/buildlet, cmd/coordinator: delete old --reverse mode All the buildlets have been updated to use --reverse-type with a host type instead of a builder type. Fixes golang/go#21260 Change-Id: I1264261f099c3686fe01455949486f523b94c6de Reviewed-on: https://go-review.googlesource.com/c/build/+/205608 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-06 23:34:49 +03:00
Notes: "run by IBM",
OwnerGithub: "mundaym",
IsReverse: true,
env: []string{"GOROOT_BOOTSTRAP=/var/buildlet/go-linux-s390x-bootstrap"},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
all: add ppc64 (big endian) builders Change-Id: I0fddda33179555a50b6f03dcff13338544b742c3 Reviewed-on: https://go-review.googlesource.com/30599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-06 23:30:58 +03:00
"host-linux-ppc64-osu": &HostConfig{
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
Notes: "Debian jessie; run by Go team on osuosl.org",
IsReverse: true,
ExpectNum: 5,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
dashboard: fix ssh username for ppc64 builders Updates golang/go#34739 Change-Id: I9e9bcf66a4e3939a68178b344672a1bab2698bb4 Reviewed-on: https://go-review.googlesource.com/c/build/+/200377 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-10-10 20:44:44 +03:00
SSHUsername: "root",
all: update linux-ppc64le builders Fixes golang/go#21189 Updates golang/go#21260 Updates golang/go#34830 (ppc64 big endian remains) Change-Id: I7ff38edb519bf4f34cd05bb795b6a41f72b10640 Reviewed-on: https://go-review.googlesource.com/c/build/+/201637 Reviewed-by: Carlos Eduardo Seo <cseo@linux.vnet.ibm.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-17 08:08:40 +03:00
HermeticReverse: false, // TODO: run in chroots with overlayfs? https://github.com/golang/go/issues/34830#issuecomment-543386764
all: add ppc64 (big endian) builders Change-Id: I0fddda33179555a50b6f03dcff13338544b742c3 Reviewed-on: https://go-review.googlesource.com/30599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-06 23:30:58 +03:00
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
"host-linux-ppc64le-osu": &HostConfig{
all: update linux-ppc64le builders Fixes golang/go#21189 Updates golang/go#21260 Updates golang/go#34830 (ppc64 big endian remains) Change-Id: I7ff38edb519bf4f34cd05bb795b6a41f72b10640 Reviewed-on: https://go-review.googlesource.com/c/build/+/201637 Reviewed-by: Carlos Eduardo Seo <cseo@linux.vnet.ibm.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-17 08:08:40 +03:00
Notes: "Debian Buster; run by Go team on osuosl.org; see x/build/env/linux-ppc64le/osuosl",
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
IsReverse: true,
ExpectNum: 5,
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
dashboard: fix ssh username for ppc64 builders Updates golang/go#34739 Change-Id: I9e9bcf66a4e3939a68178b344672a1bab2698bb4 Reviewed-on: https://go-review.googlesource.com/c/build/+/200377 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-10-10 20:44:44 +03:00
SSHUsername: "root",
all: update linux-ppc64le builders Fixes golang/go#21189 Updates golang/go#21260 Updates golang/go#34830 (ppc64 big endian remains) Change-Id: I7ff38edb519bf4f34cd05bb795b6a41f72b10640 Reviewed-on: https://go-review.googlesource.com/c/build/+/201637 Reviewed-by: Carlos Eduardo Seo <cseo@linux.vnet.ibm.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-17 08:08:40 +03:00
HermeticReverse: true,
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
},
dashboard: add linux-ppc64le POWER9 builder This change adds an extra linux-ppc64le builder running on POWER9. Fixes golang/go#30138 Change-Id: I23bdfc116bb89c6d96be330c2c87827c47705b6b Reviewed-on: https://go-review.googlesource.com/c/163757 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:05:12 +03:00
"host-linux-ppc64le-power9-osu": &HostConfig{
all: update linux-ppc64le builders Fixes golang/go#21189 Updates golang/go#21260 Updates golang/go#34830 (ppc64 big endian remains) Change-Id: I7ff38edb519bf4f34cd05bb795b6a41f72b10640 Reviewed-on: https://go-review.googlesource.com/c/build/+/201637 Reviewed-by: Carlos Eduardo Seo <cseo@linux.vnet.ibm.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-17 08:08:40 +03:00
Notes: "Debian Buster; run by Go team on osuosl.org; see x/build/env/linux-ppc64le/osuosl",
dashboard: add linux-ppc64le POWER9 builder This change adds an extra linux-ppc64le builder running on POWER9. Fixes golang/go#30138 Change-Id: I23bdfc116bb89c6d96be330c2c87827c47705b6b Reviewed-on: https://go-review.googlesource.com/c/163757 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:05:12 +03:00
IsReverse: true,
dashboard: fix POWER9 builder code generation When we moved the ppc64le builders to Docker, GOPPC64=power9 was not set for the POWER9 builders. This variable has to be passed to the environment so the compiler can generate POWER9 instructions. Fixes golang/go#36802 Change-Id: I13e99eb316e2126302e26ee771aaf8bbfbbc1c8c Reviewed-on: https://go-review.googlesource.com/c/build/+/216577 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2020-01-27 21:28:06 +03:00
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap", "GOPPC64=power9"},
dashboard: fix ssh username for ppc64le-power9 builder Updates golang/go#34739 Change-Id: I6499ff6a0e1885fad3a5f5ba7c1fa4e0113e716c Reviewed-on: https://go-review.googlesource.com/c/build/+/200398 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-10-10 22:48:15 +03:00
SSHUsername: "root",
all: update linux-ppc64le builders Fixes golang/go#21189 Updates golang/go#21260 Updates golang/go#34830 (ppc64 big endian remains) Change-Id: I7ff38edb519bf4f34cd05bb795b6a41f72b10640 Reviewed-on: https://go-review.googlesource.com/c/build/+/201637 Reviewed-by: Carlos Eduardo Seo <cseo@linux.vnet.ibm.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-17 08:08:40 +03:00
HermeticReverse: true,
dashboard: add linux-ppc64le POWER9 builder This change adds an extra linux-ppc64le builder running on POWER9. Fixes golang/go#30138 Change-Id: I23bdfc116bb89c6d96be330c2c87827c47705b6b Reviewed-on: https://go-review.googlesource.com/c/163757 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:05:12 +03:00
},
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
"host-linux-arm64-packet": &HostConfig{
env/linux-arm64: update container image from Debian Jessie to Buster Fixes golang/go#34905 Change-Id: Ib467d31b158b5ad3f4f993959fdf1a25ac91b8b1 Reviewed-on: https://go-review.googlesource.com/c/build/+/201080 Reviewed-by: Keith Randall <khr@golang.org>
2019-10-15 07:44:49 +03:00
Notes: "On 96 core packet.net host (Xenial) in Docker containers (Debian Buster); run by Go team. See x/build/env/linux-arm64/packet",
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
IsReverse: true,
HermeticReverse: true,
dashboard: restrict CPU usage on packet builders The packet arm64 build box has 96 cores, and easily gets overloaded with multiple builds running simultaneously. Restrict each container to fewer cores, and decrease the number of workers. In order to not lose some of the coverage we got from packet as a high-CPU builder, we will need to create new builder types going forward to ensure we don't lose coverage. See the issue for details. Updates golang/go#36170 Change-Id: I16478fa05258bc14ac59a1e708133ffcc4c5a664 Reviewed-on: https://go-review.googlesource.com/c/build/+/211579 Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-12-17 01:14:27 +03:00
ExpectNum: 8,
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
SSHUsername: "root",
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
},
buildlet: add an AWS buildlet client This change adds an AWS buildlet client which allows us to create EC2 instances on AWS. With this change we have also moved a portion of the gce creation logic into a helper function which allows multiple clients to use it. Metadata for the instances are stored in the user data fields. The creation of a buildlet pool and modifications to rundocker buildlet be made in order to enable this change. Updates golang/go#36841 Change-Id: Ice03e1520513d51a02b9d66542e00012453bf0d9 Reviewed-on: https://go-review.googlesource.com/c/build/+/232077 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-04 20:13:17 +03:00
"host-linux-arm64-aws": &HostConfig{
dashboard, buildenv: label EC2 instances This change adds a boolean which identified when an instance is an EC2 instance. Various EC2 related variables have been renamed to be more consistent. Updates golang/go#36841 Change-Id: Ief385bf9d41e320ebd7d31bc0108e8053f337afd Reviewed-on: https://go-review.googlesource.com/c/build/+/233801 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-13 20:27:15 +03:00
Notes: "Debian Buster, EC2 arm64 instance. See x/build/env/linux-arm64/aws",
dashboard, env/linux-arm64/aws: update EC2 ami search query This change updates which source ami is used as a base for the creation of the linux-arm64-aws builder. Packer will now search for the latest version of the arm64 Debian image to use as the source ami. For golang/go#36841 Change-Id: I036dbaf7ee65d4e7028cd6a0cdf6710bf898fdd2 Reviewed-on: https://go-review.googlesource.com/c/build/+/266697 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-10-29 00:23:17 +03:00
VMImage: "ami-03089323a1d38e652",
env/linux-arm64/aws, dashboard: update linux-arm64 builder image This updates the instruction for creating a VM image and a buildlet container image. The instructions will provide a better description for the steps required to create the images. The VM image has been updated to no longer require sudo when the docker command is executed. The host configuration has been updated with a new VM type and VM image. The rundockerbuildlet service configuration has been modified: * The memory limit has been removed because rundockerbuildlet will now use all of the available memory for buildlets running on AWS. * The service will report as running even after rundockerbuildlet has exited. Rundockerbuildlet will initiate a single buildlet and then exit. * The service will not be restarted after rundockerbuildlet has exited. Changes made to rundockerbuildlet which required these changes have been made in CL 234114. Updates golang/go#36841 Change-Id: I3ff51e414b11a2b1427cbd7c6e24e9135fc2a5c3 Reviewed-on: https://go-review.googlesource.com/c/build/+/236297 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-20 00:26:14 +03:00
ContainerImage: "gobuilder-arm64-aws:latest",
machineType: "m6g.xlarge",
dashboard, buildenv: label EC2 instances This change adds a boolean which identified when an instance is an EC2 instance. Various EC2 related variables have been renamed to be more consistent. Updates golang/go#36841 Change-Id: Ief385bf9d41e320ebd7d31bc0108e8053f337afd Reviewed-on: https://go-review.googlesource.com/c/build/+/233801 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-13 20:27:15 +03:00
isEC2: true,
buildlet: add an AWS buildlet client This change adds an AWS buildlet client which allows us to create EC2 instances on AWS. With this change we have also moved a portion of the gce creation logic into a helper function which allows multiple clients to use it. Metadata for the instances are stored in the user data fields. The creation of a buildlet pool and modifications to rundocker buildlet be made in order to enable this change. Updates golang/go#36841 Change-Id: Ice03e1520513d51a02b9d66542e00012453bf0d9 Reviewed-on: https://go-review.googlesource.com/c/build/+/232077 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-04 20:13:17 +03:00
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
buildlet: use TLS certificates for SSH connections This change adds the use of TLS certificates when connecting to a remote buildlet via port 443. This change only applies to the ConnectSSH, all other remote buildlet connections already use the TLS certificate. Fixes golang/go#41697 Change-Id: Ibc3e85edb562c42c9da2b1025f4b291b4a88deaa Reviewed-on: https://go-review.googlesource.com/c/build/+/258097 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-09-28 22:42:57 +03:00
SSHUsername: "root",
buildlet: add an AWS buildlet client This change adds an AWS buildlet client which allows us to create EC2 instances on AWS. With this change we have also moved a portion of the gce creation logic into a helper function which allows multiple clients to use it. Metadata for the instances are stored in the user data fields. The creation of a buildlet pool and modifications to rundocker buildlet be made in order to enable this change. Updates golang/go#36841 Change-Id: Ice03e1520513d51a02b9d66542e00012453bf0d9 Reviewed-on: https://go-review.googlesource.com/c/build/+/232077 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-04 20:13:17 +03:00
},
dashboard: add interim illumos builder While the work to make illumos a first class GCE guest is completed, use this interim zone provided by an illumos community member to run illumos builds. Updates golang/go#15581 Change-Id: I1784847e5407894d01ce0aadf489b38d7e5c1924 Reviewed-on: https://go-review.googlesource.com/c/build/+/201597 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-10-17 11:48:13 +03:00
"host-illumos-amd64-jclulow": &HostConfig{
Notes: "SmartOS base64@19.1.0 zone",
Owner: "josh@sysmgr.org",
OwnerGithub: "jclulow",
IsReverse: true,
ExpectNum: 1,
SSHUsername: "gobuild",
},
x/build: add Solaris amd64 release builder to dashboard Fixes golang/go#20949 Change-Id: I4f85ee76783a1240a2ebd4a2a1f167ab849fcebe Reviewed-on: https://go-review.googlesource.com/47811 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-07 23:41:20 +03:00
"host-solaris-oracle-amd64-oraclerel": &HostConfig{
Notes: "Oracle Solaris amd64 Release System",
dashboard: update Solaris owner Updates golang/go#15581 Change-Id: Idae332b234d18f0bd90eb354f611fd5a824feb85 Reviewed-on: https://go-review.googlesource.com/c/build/+/205600 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-06 19:59:20 +03:00
Owner: "",
OwnerGithub: "rorth", // https://github.com/golang/go/issues/15581#issuecomment-550368581
dashboard: add Oracle Solaris development builder Fixes golang/go#15072 Change-Id: I85677636a0f7f8b6a7b2405bb0669c39cc1da7f0 Reviewed-on: https://go-review.googlesource.com/46831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-27 20:11:15 +03:00
IsReverse: true,
ExpectNum: 1,
x/build: bootstrap tweaks for solaris builders Change bootstrap logic for solaris builders to avoid applying it to Oracle Solaris builders. Additionally, update the location of the Go bootstrap to reflect actual Oracle Solaris builder configuration. Fixes golang/go#21241 Change-Id: Ia2f21fab1581b4c195256c0dca7a87b4212798bf Reviewed-on: https://go-review.googlesource.com/52310 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-31 23:58:19 +03:00
env: []string{"GOROOT_BOOTSTRAP=/opt/golang/go-solaris-amd64-bootstrap"},
dashboard: add Oracle Solaris development builder Fixes golang/go#15072 Change-Id: I85677636a0f7f8b6a7b2405bb0669c39cc1da7f0 Reviewed-on: https://go-review.googlesource.com/46831 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-27 20:11:15 +03:00
},
dashboard: add linux-mipsle-mengzhuo builder Updates golang/go#31217 Change-Id: I7bda4f3ddf7dc76bdd974726747621f69a7acfa0 Reviewed-on: https://go-review.googlesource.com/c/build/+/191577 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-10 02:21:11 +03:00
"host-linux-mipsle-mengzhuo": &HostConfig{
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
Notes: "Loongson 3A Box hosted by Meng Zhuo; actually MIPS64 despite the name",
dashboard: add linux-mipsle-mengzhuo builder Updates golang/go#31217 Change-Id: I7bda4f3ddf7dc76bdd974726747621f69a7acfa0 Reviewed-on: https://go-review.googlesource.com/c/build/+/191577 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-10 02:21:11 +03:00
OwnerGithub: "mengzhuo",
IsReverse: true,
ExpectNum: 1,
env: []string{
"GOROOT_BOOTSTRAP=/usr/lib/golang",
"GOMIPS64=hardfloat",
},
},
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
"host-linux-mips64le-rtrk": &HostConfig{
Notes: "cavium,rhino_utm8 board hosted at RT-RK.com; quad-core cpu, 8GB of ram and 240GB ssd disks.",
OwnerGithub: "bogojevic", // and @milanknezevic. https://github.com/golang/go/issues/31217#issuecomment-547004892
IsReverse: true,
ExpectNum: 1,
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap",
},
},
"host-linux-mips64-rtrk": &HostConfig{
Notes: "cavium,rhino_utm8 board hosted at RT-RK.com; quad-core cpu, 8GB of ram and 240GB ssd disks.",
OwnerGithub: "bogojevic", // and @milanknezevic. https://github.com/golang/go/issues/31217#issuecomment-547004892
IsReverse: true,
ExpectNum: 1,
env: []string{
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap",
},
},
dashboard,env: move iOS builders to GOOS=ios Updates golang/go#41610 Change-Id: Ie8cb3294da5566e09fc06c01e679b8b773a439cf Reviewed-on: https://go-review.googlesource.com/c/build/+/259358 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Go Bot <gobot@golang.org> Trust: Elias Naur <mail@eliasnaur.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-10-03 12:37:07 +03:00
"host-ios-arm64-corellium-ios": &HostConfig{
dashboard: reduce expected number of corellium builders to 2 each According to https://golang.org/issue/40523#issuecomment-676546824, there are only 2 Android corellium hosts and 2 iOS corellium hosts. Also add links to issues with more information to the builder notes. Updates golang/go#40523. Updates golang/go#31722. Change-Id: Ibabccebfcc6735cba867cccf109c71607223d1a2 Reviewed-on: https://go-review.googlesource.com/c/build/+/249337 Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-08-19 20:15:55 +03:00
Notes: "Virtual iOS devices hosted by Zenly on Corellium; see issues 31722 and 40523",
dashboard: add Zenly's virtual iOS builders Fixes golang/go#31722 Change-Id: Ia7f0d3f2e1a25000c42a10ff58d7e91d384b1219 Reviewed-on: https://go-review.googlesource.com/c/build/+/174797 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 21:33:26 +03:00
OwnerGithub: "znly",
IsReverse: true,
env/corellium: update Corellium builder setup scripts Corellium recently wiped the existing virtual iPhone and Android builders because of a cloud provider switch. This change reflects the updates to the setup scripts matching newer Corellium software. Bump the expected builders numbers to reflect the increased capacity. Change-Id: I93196dd22b81e66cb2cad82b1ef3dcff64e852d8 Reviewed-on: https://go-review.googlesource.com/c/build/+/266377 Run-TryBot: Elias Naur <mail@eliasnaur.com> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Elias Naur <mail@eliasnaur.com>
2020-10-29 21:04:57 +03:00
ExpectNum: 3,
dashboard: add Zenly's virtual iOS builders Fixes golang/go#31722 Change-Id: Ia7f0d3f2e1a25000c42a10ff58d7e91d384b1219 Reviewed-on: https://go-review.googlesource.com/c/build/+/174797 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 21:33:26 +03:00
env: []string{
dashboard,env: move iOS builders to GOOS=ios Updates golang/go#41610 Change-Id: Ie8cb3294da5566e09fc06c01e679b8b773a439cf Reviewed-on: https://go-review.googlesource.com/c/build/+/259358 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Go Bot <gobot@golang.org> Trust: Elias Naur <mail@eliasnaur.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-10-03 12:37:07 +03:00
"GOROOT_BOOTSTRAP=/var/mobile/go-ios-arm64-bootstrap",
dashboard: add Zenly's virtual iOS builders Fixes golang/go#31722 Change-Id: Ia7f0d3f2e1a25000c42a10ff58d7e91d384b1219 Reviewed-on: https://go-review.googlesource.com/c/build/+/174797 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 21:33:26 +03:00
},
},
dashboard: add Zenly's virtual Android builders Change-Id: Id0be99f343a819c19b0004dc4160175c9bb0ace9 Reviewed-on: https://go-review.googlesource.com/c/build/+/174798 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 23:28:43 +03:00
"host-android-arm64-corellium-android": &HostConfig{
dashboard: reduce expected number of corellium builders to 2 each According to https://golang.org/issue/40523#issuecomment-676546824, there are only 2 Android corellium hosts and 2 iOS corellium hosts. Also add links to issues with more information to the builder notes. Updates golang/go#40523. Updates golang/go#31722. Change-Id: Ibabccebfcc6735cba867cccf109c71607223d1a2 Reviewed-on: https://go-review.googlesource.com/c/build/+/249337 Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-08-19 20:15:55 +03:00
Notes: "Virtual Android devices hosted by Zenly on Corellium; see issues 31722 and 40523",
dashboard: add Zenly's virtual Android builders Change-Id: Id0be99f343a819c19b0004dc4160175c9bb0ace9 Reviewed-on: https://go-review.googlesource.com/c/build/+/174798 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 23:28:43 +03:00
OwnerGithub: "znly",
IsReverse: true,
env/corellium: update Corellium builder setup scripts Corellium recently wiped the existing virtual iPhone and Android builders because of a cloud provider switch. This change reflects the updates to the setup scripts matching newer Corellium software. Bump the expected builders numbers to reflect the increased capacity. Change-Id: I93196dd22b81e66cb2cad82b1ef3dcff64e852d8 Reviewed-on: https://go-review.googlesource.com/c/build/+/266377 Run-TryBot: Elias Naur <mail@eliasnaur.com> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Elias Naur <mail@eliasnaur.com>
2020-10-29 21:04:57 +03:00
ExpectNum: 3,
dashboard: add Zenly's virtual Android builders Change-Id: Id0be99f343a819c19b0004dc4160175c9bb0ace9 Reviewed-on: https://go-review.googlesource.com/c/build/+/174798 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 23:28:43 +03:00
env: []string{
"GOROOT_BOOTSTRAP=/data/data/com.termux/files/home/go-android-arm64-bootstrap",
},
},
dashboard: add aix-ppc64 builder This is a starting point for adding an aix-ppc64 reverse builder. There can be further adjustments made to it as needed. Updates golang/go#27160. Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508 Reviewed-on: https://go-review.googlesource.com/131335 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-24 21:41:05 +03:00
"host-aix-ppc64-osuosl": &HostConfig{
Notes: "AIX 7.2 VM on OSU; run by Tony Reix",
OwnerGithub: "trex58",
IsReverse: true,
ExpectNum: 1,
env: []string{"GOROOT_BOOTSTRAP=/opt/freeware/lib/golang"},
},
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
"host-android-amd64-emu": &HostConfig{
Notes: "Debian Buster w/ Android SDK + emulator (use nested virt)",
env/android-amd64-emu: enable gomobile device tests By adding gradle and Android build tools to PATH, setting ANDROID_HOME, and adding the OpenAL headers to CGO_CFLAGS. Change-Id: Ifeaa1b82f4108afc3787eee4449b3317185bbaec Reviewed-on: https://go-review.googlesource.com/c/build/+/164658 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-01 13:21:20 +03:00
ContainerImage: "android-amd64-emu:bff27c0c9263",
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
KonletVMImage: "android-amd64-emu",
NestedVirt: true,
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
SSHUsername: "root",
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
}
dashboard, cmd/coordinator: move cross compilation config into dashboard package The dashboard package has been gradually absorbing all configuration, keeping cmd/coordinator only about mechanics. But we'd missed this cross compilation bit in earlier cleanups. Move it, and add some tests & docs while I'm at it. Also, disable the linux-arm-nativemake builder by default. It's redundant with linux-arm. They used to be more different when we always cross compiled for linux-arm, but I guess that got disabled at some point. At this poin it serves no purpose except as an opt-in slowbot if the user really wants to run make.bash on scaleway. Change-Id: Iac683ccf4d30214330480f9db39f131263b1bf8a Reviewed-on: https://go-review.googlesource.com/c/build/+/202020 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-10-19 05:49:53 +03:00
// CrossCompileConfig describes how to cross-compile a build on a
// faster host.
type CrossCompileConfig struct {
// CompileHostType is the host type to use for compilation
CompileHostType string
// CCForTarget is the CC_FOR_TARGET environment variable.
CCForTarget string
// GOARM is any GOARM= environment variable.
GOARM string
// AlwaysCrossCompile controls whether this builder always
// cross compiles. Otherwise it's only done for trybot runs.
AlwaysCrossCompile bool
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func init() {
for key, c := range Hosts {
if key == "" {
panic("empty string key in Hosts")
}
if c.HostType == "" {
c.HostType = key
}
if c.HostType != key {
panic(fmt.Sprintf("HostType %q != key %q", c.HostType, key))
}
nSet := 0
env/linux-arm64/aws, dashboard: update linux-arm64 builder image This updates the instruction for creating a VM image and a buildlet container image. The instructions will provide a better description for the steps required to create the images. The VM image has been updated to no longer require sudo when the docker command is executed. The host configuration has been updated with a new VM type and VM image. The rundockerbuildlet service configuration has been modified: * The memory limit has been removed because rundockerbuildlet will now use all of the available memory for buildlets running on AWS. * The service will report as running even after rundockerbuildlet has exited. Rundockerbuildlet will initiate a single buildlet and then exit. * The service will not be restarted after rundockerbuildlet has exited. Changes made to rundockerbuildlet which required these changes have been made in CL 234114. Updates golang/go#36841 Change-Id: I3ff51e414b11a2b1427cbd7c6e24e9135fc2a5c3 Reviewed-on: https://go-review.googlesource.com/c/build/+/236297 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-20 00:26:14 +03:00
if c.VMImage != "" && !c.isEC2 {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
nSet++
}
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
if c.ContainerImage != "" {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
nSet++
}
if c.IsReverse {
nSet++
}
if nSet != 1 {
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
panic(fmt.Sprintf("exactly one of VMImage, ContainerImage, IsReverse must be set for host %q; got %v", key, nSet))
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
}
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
if c.buildletURLTmpl == "" && (c.VMImage != "" || c.ContainerImage != "") {
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
panic(fmt.Sprintf("missing buildletURLTmpl for host type %q", key))
}
}
}
// A HostConfig describes the available ways to obtain buildlets of
dashboard, app/appengine: set a known issue for staticlockranking builder The linux-amd64-staticlockranking builder was added pre-emptively so that it could be tested and used during the development of a CL that implements a new feature. It's expected to fail until that work is completed. Mark it as a builder with a known issue, golang/go#38029. The new BuildConfig.KnownIssue field can be used by builders being added in the future so that if they fail at first, it doesn't take away time for people looking at build.golang.org, which we always aim to have as green as possible¹. ¹ https://groups.google.com/d/msg/golang-dev/y0yM_Xi665Q/hUpBiUPiCAAJ Fixes golang/go#38283 For golang/go#38029 For golang/go#37937 For golang/go#11811 Change-Id: Iba1606c7021ffa7e67ddbaae2fc6b65f4e46ab34 Reviewed-on: https://go-review.googlesource.com/c/build/+/227379 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-07 00:34:17 +03:00
// different types. Some host configs can serve multiple
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
// builders. For example, a host config of "host-linux-jessie" can
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// serve linux-amd64, linux-amd64-race, linux-386, linux-386-387, etc.
type HostConfig struct {
// HostType is the unique name of this host config. It is also
// the key in the Hosts map.
HostType string
// buildletURLTmpl is the URL "template" ($BUCKET is auto-expanded)
// for the URL to the buildlet binary.
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
// This field is required for VM and Container builders. It's not
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// needed for reverse buildlets because in that case, the buildlets
// are already running and their stage0 should know how to update it
// it automatically.
buildletURLTmpl string
dashboard: add host-linux-arm64-aws builder Add the host-linux-arm64-aws builder. The HostConfig IsVM method has been modified to support the concept that an EC2 VM can be created which runs the buildlet binary in a container. This configuration would have both a VM image and a container image set in the host configuration. For golang/go#36841 Change-Id: I092b48b1dda24d24bcd0b10dff0b2126f5f43017 Reviewed-on: https://go-review.googlesource.com/c/build/+/249118 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-18 20:32:05 +03:00
// Exactly 1 of these must be set (with the exception of EC2 instances).
// An EC2 instance may run a container inside a VM. In that case, a VMImage
// and ContainerImage will both be set.
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
VMImage string // e.g. "openbsd-amd64-60"
ContainerImage string // e.g. "linux-buildlet-std:latest" (suffix after "gcr.io/<PROJ>/")
IsReverse bool // if true, only use the reverse buildlet pool
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// GCE options, if VMImage != ""
buildlet, dashboard: add min CPU platform knob, require Skylake for OpenBSD I thought this would be enough for OpenBSD to select the TSC on its own without being forced to (as in CL 160319), but apparently it is not: https://github.com/golang/go/issues/29223#issuecomment-459035482 So it seems like we want both this CL and CL 160319. Updates golang/go#29223 Change-Id: I0a092d62881d8dcce0ef1129d8d32d8f4025b6ac Reviewed-on: https://go-review.googlesource.com/c/160457 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-30 20:13:07 +03:00
machineType string // optional GCE instance type
RegularDisk bool // if true, use spinning disk instead of SSD
MinCPUPlatform string // optional; https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
dashboard, buildenv: label EC2 instances This change adds a boolean which identified when an instance is an EC2 instance. Various EC2 related variables have been renamed to be more consistent. Updates golang/go#36841 Change-Id: Ief385bf9d41e320ebd7d31bc0108e8053f337afd Reviewed-on: https://go-review.googlesource.com/c/build/+/233801 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-13 20:27:15 +03:00
// EC2 options
isEC2 bool // if true, the instance is configured to run on EC2
cmd/coordinator: clean up reverse buildlet code, export status JSON Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-24 20:50:49 +03:00
// ReverseOptions:
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
ExpectNum int // expected number of reverse buildlets of this type
HermeticReverse bool // whether reverse buildlet has fresh env per conn
cmd/coordinator: clean up reverse buildlet code, export status JSON Will be used for dynamic creation/destruction of Mac VMs in subsequent CL. Updates golang/go#9495 (Mac virtualization) Updates golang/go#15760 (monitoring) Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149 Reviewed-on: https://go-review.googlesource.com/37457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-24 20:50:49 +03:00
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
// Container image options, if ContainerImage != "":
env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx This adds scripts to create a new builder host image that acts like Container-Optimized OS (has docker, runs konlet on startup) but with a Debian 9 kernel + userspace that permits KVM for nested virtualization. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884 Reviewed-on: https://go-review.googlesource.com/c/163301 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-22 00:10:54 +03:00
NestedVirt bool // container requires VMX nested virtualization
KonletVMImage string // optional VM image (containing konlet) to use instead of default
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
// Optional base env. GOROOT_BOOTSTRAP should go here if the buildlet
// has Go 1.4+ baked in somewhere.
env []string
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
// These template URLs may contain $BUCKET which is expanded to the
// relevant Cloud Storage bucket as specified by the build environment.
goBootstrapURLTmpl string // optional URL to a built Go 1.4+ tar.gz
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
Owner string // optional email of owner; "bradfitz@golang.org", empty means golang-dev
OwnerGithub string // optional GitHub username of owner
Notes string // notes for humans
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
SSHUsername string // username to ssh as, empty means not supported
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
}
// A BuildConfig describes how to run a builder.
type BuildConfig struct {
// Name is the unique name of the builder, in the form of
// "GOOS-GOARCH" or "GOOS-GOARCH-suffix". For example,
// "darwin-386", "linux-386-387", "linux-amd64-race". Some
// suffixes are well-known and carry special meaning, such as
// "-race".
Name string
// HostType is the required key into the Hosts map, describing
// the type of host this build will run on.
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
// For example, "host-linux-jessie".
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
HostType string
dashboard, app/appengine: set a known issue for staticlockranking builder The linux-amd64-staticlockranking builder was added pre-emptively so that it could be tested and used during the development of a CL that implements a new feature. It's expected to fail until that work is completed. Mark it as a builder with a known issue, golang/go#38029. The new BuildConfig.KnownIssue field can be used by builders being added in the future so that if they fail at first, it doesn't take away time for people looking at build.golang.org, which we always aim to have as green as possible¹. ¹ https://groups.google.com/d/msg/golang-dev/y0yM_Xi665Q/hUpBiUPiCAAJ Fixes golang/go#38283 For golang/go#38029 For golang/go#37937 For golang/go#11811 Change-Id: Iba1606c7021ffa7e67ddbaae2fc6b65f4e46ab34 Reviewed-on: https://go-review.googlesource.com/c/build/+/227379 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-07 00:34:17 +03:00
// KnownIssue is a non-zero golang.org/issue/nnn number for a builder
dashboard: create misc-compile-ios builder Create the builder as a post-submit builder with a non-zero KnownIssue value first. Once we know it works well, we'll have the confidence to be able to safely promote it to a normal TryBot, without the risk of causing disruption for contributors. For golang/go#41610. Change-Id: If871addf325b7728109afbe094b72d40e27ceec1 Reviewed-on: https://go-review.googlesource.com/c/build/+/257617 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2020-09-25 18:52:04 +03:00
// that may fail due to a known issue, such as because it is a new
// builder still in development/testing, or because the feature
dashboard, app/appengine: set a known issue for staticlockranking builder The linux-amd64-staticlockranking builder was added pre-emptively so that it could be tested and used during the development of a CL that implements a new feature. It's expected to fail until that work is completed. Mark it as a builder with a known issue, golang/go#38029. The new BuildConfig.KnownIssue field can be used by builders being added in the future so that if they fail at first, it doesn't take away time for people looking at build.golang.org, which we always aim to have as green as possible¹. ¹ https://groups.google.com/d/msg/golang-dev/y0yM_Xi665Q/hUpBiUPiCAAJ Fixes golang/go#38283 For golang/go#38029 For golang/go#37937 For golang/go#11811 Change-Id: Iba1606c7021ffa7e67ddbaae2fc6b65f4e46ab34 Reviewed-on: https://go-review.googlesource.com/c/build/+/227379 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-07 00:34:17 +03:00
// or port that it's meant to test hasn't been added yet, etc.
//
// A non-zero value here means that failures on this builder should not
// be considered a serious regression and don't need investigation beyond
// what is already in scope of the listed issue.
KnownIssue int
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes string // notes for humans
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// tryBot optionally specifies a policy func for whether trybots are enabled.
// nil means off. Even if tryBot returns true, BuildConfig.BuildsRepo must also
// return true. See the implementation of BuildConfig.BuildsRepoTryBot.
// The proj is "go", "net", etc. The branch is proj's branch.
// The goBranch is the same as branch for proj "go", else it's the go branch
// ("master, "release-branch.go1.12", etc).
tryBot func(proj, branch, goBranch string) bool
tryOnly bool // only used for trybots, and not regular builds
CompileOnly bool // if true, compile tests, but don't run them
FlakyNet bool // network tests are flaky (try anyway, but ignore some failures)
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
// buildsRepo optionally specifies whether this builder does
// builds (of any type) for the given repo ("go", "net", etc.)
// and its branch ("master", "release-branch.go1.12", "dev.link", etc.).
// goBranch is the branch of "go" to build against. If repo == "go",
// goBranch == branch.
dashboard: add linux-amd64-staticlockranking builder This is a new linux/amd64 GCE builder for the purpose of testing the GOEXPERIMENT=staticlockranking configuration which is landing to master branch soon. It runs only on the main Go repository master branch and release branches 1.15 and newer. No golang.org/x repos. Not included in trybots by default, but can be requested via SlowBots by saying TRY=linux-amd64-staticlockranking. Also expand documentation of BuildConfig.buildsRepo field to make it more clear how to use it. A common mistake for custom buildsRepo functions that are added is to create a custom policy with a minor adjustment and forgetting to preserve the rest of the default build repo policy. Fixes golang/go#37937 Change-Id: Ic56f82e9ede30cb6fe8f6cc7d0282d6a0a12c1b8 Reviewed-on: https://go-review.googlesource.com/c/build/+/224078 Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-03-19 17:34:11 +03:00
//
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
// If nil, a default set of repos as reported by buildRepoByDefault
// is built. See buildsRepoAtAll method for details.
dashboard: add linux-amd64-staticlockranking builder This is a new linux/amd64 GCE builder for the purpose of testing the GOEXPERIMENT=staticlockranking configuration which is landing to master branch soon. It runs only on the main Go repository master branch and release branches 1.15 and newer. No golang.org/x repos. Not included in trybots by default, but can be requested via SlowBots by saying TRY=linux-amd64-staticlockranking. Also expand documentation of BuildConfig.buildsRepo field to make it more clear how to use it. A common mistake for custom buildsRepo functions that are added is to create a custom policy with a minor adjustment and forgetting to preserve the rest of the default build repo policy. Fixes golang/go#37937 Change-Id: Ic56f82e9ede30cb6fe8f6cc7d0282d6a0a12c1b8 Reviewed-on: https://go-review.googlesource.com/c/build/+/224078 Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-03-19 17:34:11 +03:00
//
// To implement a minor change to the default policy, create a
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
// function that uses buildRepoByDefault. For example:
dashboard: add linux-amd64-staticlockranking builder This is a new linux/amd64 GCE builder for the purpose of testing the GOEXPERIMENT=staticlockranking configuration which is landing to master branch soon. It runs only on the main Go repository master branch and release branches 1.15 and newer. No golang.org/x repos. Not included in trybots by default, but can be requested via SlowBots by saying TRY=linux-amd64-staticlockranking. Also expand documentation of BuildConfig.buildsRepo field to make it more clear how to use it. A common mistake for custom buildsRepo functions that are added is to create a custom policy with a minor adjustment and forgetting to preserve the rest of the default build repo policy. Fixes golang/go#37937 Change-Id: Ic56f82e9ede30cb6fe8f6cc7d0282d6a0a12c1b8 Reviewed-on: https://go-review.googlesource.com/c/build/+/224078 Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-03-19 17:34:11 +03:00
//
// buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
// b := buildRepoByDefault(repo)
dashboard: add linux-amd64-staticlockranking builder This is a new linux/amd64 GCE builder for the purpose of testing the GOEXPERIMENT=staticlockranking configuration which is landing to master branch soon. It runs only on the main Go repository master branch and release branches 1.15 and newer. No golang.org/x repos. Not included in trybots by default, but can be requested via SlowBots by saying TRY=linux-amd64-staticlockranking. Also expand documentation of BuildConfig.buildsRepo field to make it more clear how to use it. A common mistake for custom buildsRepo functions that are added is to create a custom policy with a minor adjustment and forgetting to preserve the rest of the default build repo policy. Fixes golang/go#37937 Change-Id: Ic56f82e9ede30cb6fe8f6cc7d0282d6a0a12c1b8 Reviewed-on: https://go-review.googlesource.com/c/build/+/224078 Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-03-19 17:34:11 +03:00
// // ... modify b from the default value as needed ...
// return b
// }
//
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo func(repo, branch, goBranch string) bool
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
// MinimumGoVersion optionally specifies the minimum Go version
// this builder is allowed to use. It can be useful for skipping
// builders that are too new and no longer support some supported
// Go versions. It doesn't need to be set for builders that support
// all supported Go versions.
//
// Note: This field currently has effect on trybot runs only.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
//
// TODO: unexport this and make buildsRepoAtAll return false on too-old
// of repos. The callers in coordinator will need updating.
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
MinimumGoVersion types.MajorMinor
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
// SkipSnapshot, if true, means to not fetch a tarball
// snapshot of the world post-make.bash from the buildlet (and
// thus to not write it to Google Cloud Storage). This is
// incompatible with sharded tests, and should only be used
// for very slow builders or networks, unable to transfer
// the tarball in under ~5 minutes.
SkipSnapshot bool
cmd/coordinator: run benchmarks on try work Benchmarks are treated as unit tests and distributed to the test helpers, which allows them to fit in our 5m trybot budget. Currently we only run the go1 and x/benchmarks. Running package benchmarks is a TODO. This feature is disabled by default, and is enabled by the "farmer-run-bench" project attribute. Updates golang/go#19178 Updates golang/go#19871 Change-Id: I9c3a14da60c3662e7e2cb4e71953060915cc4364 Reviewed-on: https://go-review.googlesource.com/38306 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-15 23:18:30 +03:00
// RunBench causes the coordinator to run benchmarks on this buildlet type.
RunBench bool
dashboard: add a make.bash-only builder on real ARM hardware This is a new builder in prep for the change to the "linux-arm" builder where the GOARCH=arm make.bash will be cross-compiled from a Kubernetes container on fast hardware. Updates golang/go#17105 (cross-compile ARM builders' make.bash) Updates golang/go#17104 (5 minute trybots) Change-Id: Icfd2644d77639f731151abe54839322960418254 Reviewed-on: https://go-review.googlesource.com/29670 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 02:29:31 +03:00
// StopAfterMake causes the build to stop after the make
// script completes, returning its result as the result of the
// whole build. It does not run or compile any of the tests,
// nor does it write a snapshot of the world to cloud
// storage. This option is only supported for builders whose
// BuildConfig.SplitMakeRun returns true.
StopAfterMake bool
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// needsGoProxy is whether this builder should have GOPROXY set.
// Currently this is only for the longtest builder, which needs
// to run cmd/go tests fetching from the network.
needsGoProxy bool
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
// InstallRacePackages controls which packages to "go install
// -race <pkgs>" after running make.bash (or equivalent). If
// the builder ends in "-race", the default if non-nil is just
// "std".
InstallRacePackages []string
cmd/coordinator: move git ancestor checking before buildlet creation, generalize This also adds GoDeps to the builders config file and adds a dependency for the ssacheck builder, which currently fails on older release branches with: > Failed on linux-amd64-ssacheck: > https://storage.googleapis.com/go-build-log/1255edaf/linux-amd64-ssacheck_18f7e015.log > > ... > compile: unknown debug key -d dclstack Change-Id: I4ad0a298705feed56f467f3c8d9488f8be2e4af5 Reviewed-on: https://go-review.googlesource.com/43774 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-20 00:23:14 +03:00
// GoDeps is a list of of git sha1 commits that must be in the
// commit to be tested's history. If absent, this builder is
// not run for that commit.
GoDeps []string
dashboard, cmd/coordinator: move cross compilation config into dashboard package The dashboard package has been gradually absorbing all configuration, keeping cmd/coordinator only about mechanics. But we'd missed this cross compilation bit in earlier cleanups. Move it, and add some tests & docs while I'm at it. Also, disable the linux-arm-nativemake builder by default. It's redundant with linux-arm. They used to be more different when we always cross compiled for linux-arm, but I guess that got disabled at some point. At this poin it serves no purpose except as an opt-in slowbot if the user really wants to run make.bash on scaleway. Change-Id: Iac683ccf4d30214330480f9db39f131263b1bf8a Reviewed-on: https://go-review.googlesource.com/c/build/+/202020 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-10-19 05:49:53 +03:00
// CrossCompileConfig optionally specifies whether and how
// this build is cross compiled.
CrossCompileConfig *CrossCompileConfig
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
// distTestAdjust optionally specifies a function that can
// adjust the cmd/dist test policy for this builder.
//
// The BuildConfig.ShouldRunDistTest method implements the
// default cmd/dist test policy, and then calls distTestAdjust
// to adjust that decision further, if distTestAdjust is not nil.
//
// The initial value of the run parameter is what the default
// policy said. The returned value from distTestAdjust is what
// BuildConfig.ShouldRunDistTest reports to the caller.
//
// For example:
//
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
// distTestAdjust: func(run bool, distTest string, isNormalTry bool) bool {
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
// // ... modify run from the initial value as needed ...
// return run
// }
//
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
distTestAdjust func(run bool, distTest string, isNormalTry bool) bool
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
// numTestHelpers is the number of _additional_ buildlets
// past the first one to help out with sharded tests.
dashboard: try to speed up pre-submit longtest builders The longtest builders are currently primarily post-submit builders, where it's okay for them to be as slow as they need to be in order to provide additional test coverage. In this context, whether they take 40 minutes or 50 makes little difference. The longtest builders are also sometimes requested via SlowBots for changes that are riskier than usual, or otherwise desire additional coverage beyond the normal TryBots. They're also always enabled for CLs to release branches. In such contexts, speeding up SlowBot runs from 40 minutes to 20 or less would be appreciated and in turn help people use longtest SlowBots more frequently. Longtest builders are already configured to use sharded tests. Configure them to use additional helpers to speed up test execution. Try out 3, 5, and 9 helpers to see how much it helps before settling. For golang/go#37439. Change-Id: I425bc0257b7a54bb32c0eb1719fea7ba3f4fd461 Reviewed-on: https://go-review.googlesource.com/c/build/+/268037 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-11-06 15:55:36 +03:00
// For TryBots and SlowBots, the numTryHelpers value is used,
// unless it's zero, in which case numTestHelpers is used.
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
numTestHelpers int
dashboard: try to speed up pre-submit longtest builders The longtest builders are currently primarily post-submit builders, where it's okay for them to be as slow as they need to be in order to provide additional test coverage. In this context, whether they take 40 minutes or 50 makes little difference. The longtest builders are also sometimes requested via SlowBots for changes that are riskier than usual, or otherwise desire additional coverage beyond the normal TryBots. They're also always enabled for CLs to release branches. In such contexts, speeding up SlowBot runs from 40 minutes to 20 or less would be appreciated and in turn help people use longtest SlowBots more frequently. Longtest builders are already configured to use sharded tests. Configure them to use additional helpers to speed up test execution. Try out 3, 5, and 9 helpers to see how much it helps before settling. For golang/go#37439. Change-Id: I425bc0257b7a54bb32c0eb1719fea7ba3f4fd461 Reviewed-on: https://go-review.googlesource.com/c/build/+/268037 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-11-06 15:55:36 +03:00
numTryTestHelpers int // For TryBots/SlowBots. If 0, numTestHelpers is used.
cmd/coordinator: test sharding, better status, logs Also gomote updates which came about during the process of developing and debugging this. Change-Id: Ia53d674118a6b99bcdda7062d3b7161279b6ad52 Reviewed-on: https://go-review.googlesource.com/10463 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-28 07:51:25 +03:00
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
env []string // extra environment ("key=value") pairs
allScriptArgs []string
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
testHostConf *HostConfig // override HostConfig for testing, at least for now
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// Env returns the environment variables this builder should run with.
cmd/coordinator: set GO_BUILDER_NAME env var when exec'ing on buildlet This will allow tests to identify that they are running on the continuous build infrastructure and run extra and/or different tests. Fixes golang/go#11346 Change-Id: Iafe25babc764b7e012c05f0b62331b11690bf194 Reviewed-on: https://go-review.googlesource.com/13120 Reviewed-by: Russ Cox <rsc@golang.org>
2015-08-04 05:19:02 +03:00
func (c *BuildConfig) Env() []string {
dashboard: mark some builders as having flaky networks Follow-up to https://golang.org/cl/22852 Change-Id: I5f0cd25248a8892b7d1b0f14e87b12b8b9a8da3a Reviewed-on: https://go-review.googlesource.com/22883 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2016-05-06 21:55:26 +03:00
env := []string{"GO_BUILDER_NAME=" + c.Name}
if c.FlakyNet {
env = append(env, "GO_BUILDER_FLAKY_NET=1")
}
dashboard: use a more consistent definition for longtest builders Previously, it was possible to define a builder whose IsLongTest method would report positively, such that it'd test golang.org/x repos in long test mode, but the main Go repository in short test mode. This would be the case for any builder with "-longtest" suffix if it did not manually include GO_TEST_SHORT=0 in its environment configuration. It's unlikely we would want to do that intentionally, so this refactor makes such misconfiguration less likely by inserting the GO_TEST_SHORT environment variable assignment into the output from Env automatically. Now, a longtest builder is defined in one consistent way: it must have a "-longtest" suffix, so that the IsLongTest method reports positively. For golang/go#39054. For golang/go#29252. For golang/go#12508. Change-Id: Ic24df7b3b7de7db728bba6dc6ad4dd38a2e61e82 Reviewed-on: https://go-review.googlesource.com/c/build/+/233901 Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-13 21:30:25 +03:00
if c.IsLongTest() {
// Set a private hook in cmd/dist to run main Go repository tests
// without the default -short flag. See golang.org/issue/12508.
env = append(env, "GO_TEST_SHORT=0")
}
cmd/coordinator: finish the scheduler code, at least mostly Optimizations and tuning remain, but this should be tons better than what we had before (random). Updates golang/go#19178 Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de Reviewed-on: https://go-review.googlesource.com/c/build/+/205078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-30 06:59:13 +03:00
env = append(env, c.HostConfig().env...)
dashboard: mark some builders as having flaky networks Follow-up to https://golang.org/cl/22852 Change-Id: I5f0cd25248a8892b7d1b0f14e87b12b8b9a8da3a Reviewed-on: https://go-review.googlesource.com/22883 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2016-05-06 21:55:26 +03:00
return append(env, c.env...)
cmd/coordinator: set GO_BUILDER_NAME env var when exec'ing on buildlet This will allow tests to identify that they are running on the continuous build infrastructure and run extra and/or different tests. Fixes golang/go#11346 Change-Id: Iafe25babc764b7e012c05f0b62331b11690bf194 Reviewed-on: https://go-review.googlesource.com/13120 Reviewed-by: Russ Cox <rsc@golang.org>
2015-08-04 05:19:02 +03:00
}
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// ModulesEnv returns the extra module-specific environment variables
// to append to this builder as a function of the repo being built
// ("go", "oauth2", "net", etc).
func (c *BuildConfig) ModulesEnv(repo string) (env []string) {
dashboard: set the correct module proxy for EC2 builders This change sets the default module proxy address for EC2 builders. EC2 builders should have the same proxy address set as reverse builders. Fixes golang/go#40914 Change-Id: I4de2ed178170f60fdbe4e2f9aa6f024bb43ee020 Reviewed-on: https://go-review.googlesource.com/c/build/+/249577 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-20 18:53:58 +03:00
// EC2 and reverse builders should set the public module proxy
// address instead of the internal proxy.
if (c.HostConfig().isEC2 || c.IsReverse()) && repo != "go" {
cmd/coordinator: use proxy.golang.org instead of private Athens Fixes golang/go#31770 Change-Id: Ia21b3c835b3692351d83e00b9a8cb73db2103192 Reviewed-on: https://go-review.googlesource.com/c/build/+/175057 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-02 23:32:40 +03:00
env = append(env, "GOPROXY=https://proxy.golang.org")
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
}
switch repo {
dashboard: set GOPROXY=off for the "go" repo to catch more network usage Updates golang/go#30612 Updates golang/go#30760 Change-Id: I296f4f7a163c1bdd59e839229c0961b21f26da89 Reviewed-on: https://go-review.googlesource.com/c/build/+/167039 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 07:44:17 +03:00
case "go":
if !c.OutboundNetworkAllowed() {
env = append(env, "GOPROXY=off")
}
dashboard: set GO111MODULE=on for x/perf CL 191378 removed some +build appengine tags from ./analysis/appengine, so those packages now require GO111MODULE=on in order to resolve the dependency on google.golang.org/appengine. Fixes golang/go#33952 Change-Id: I20c9ba3f42d488265726fda9b1c790e48a1433b3 Reviewed-on: https://go-review.googlesource.com/c/build/+/192333 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-08-29 21:26:25 +03:00
case "oauth2", "build", "perf", "website":
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
env = append(env, "GO111MODULE=on")
}
dashboard, cmd/coordinator: add controls to skip subrepo dirs in GOPATH mode This change adds controls to the dashboard package that makes it possible to skip certain directories in subrepos from being tested in GOPATH mode. It uses this control mechanism to skip the gopls directory in x/tools from being tested in GOPATH mode. That directory and everything inside is developed with support for module mode only. All other packages in x/tools continue to be supported both in module and GOPATH modes for the current and previous releases, as per release policy¹. ¹ https://golang.org/doc/devel/release.html#policy Fixes golang/go#34190 Change-Id: Icdb4d0e1419f69c84c9a3f9a33727a09a35599f4 Reviewed-on: https://go-review.googlesource.com/c/build/+/194397 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-09-09 23:48:08 +03:00
return env
}
// ShouldTestPackageInGOPATHMode is used to control whether the package
// with the specified import path should be tested in GOPATH mode.
//
// When running tests for all golang.org/* repositories in GOPATH mode,
// this method is called repeatedly with the full import path of each
// package that is found and is being considered for testing in GOPATH
// mode. It's not used and has no effect on import paths in the main
// "go" repository. It has no effect on tests done in module mode.
//
// When considering making changes here, keep the release policy in mind:
//
// https://golang.org/doc/devel/release.html#policy
//
func (*BuildConfig) ShouldTestPackageInGOPATHMode(importPath string) bool {
if importPath == "golang.org/x/tools/gopls" ||
strings.HasPrefix(importPath, "golang.org/x/tools/gopls/") {
// Don't test golang.org/x/tools/gopls/... in GOPATH mode.
return false
}
dashboard: skip testing x/net/http2/h2demo in GOPATH mode The h2demo command is an internal command that we use to powers the https://http2.golang.org server, it is not a package that others import. It was never tested before golang.org/issue/34361 because it had a +build h2demo constraint. But that build constraint is being removed, so explicitly skip testing it in GOPATH mode. The package is supported only in module mode now, since it requires third-party dependencies. Updates golang/go#34361 Change-Id: Ia3870baa10688b83f8400b1a5e13824a1c1b3d3b Reviewed-on: https://go-review.googlesource.com/c/build/+/196097 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-09-18 03:44:48 +03:00
if importPath == "golang.org/x/net/http2/h2demo" {
// Don't test golang.org/x/net/http2/h2demo in GOPATH mode.
//
// It was never tested before golang.org/issue/34361 because it
// had a +build h2demo constraint. But that build constraint is
// being removed, so explicitly skip testing it in GOPATH mode.
//
// The package is supported only in module mode now, since
// it requires third-party dependencies.
return false
}
dashboard, cmd/coordinator: add controls to skip subrepo dirs in GOPATH mode This change adds controls to the dashboard package that makes it possible to skip certain directories in subrepos from being tested in GOPATH mode. It uses this control mechanism to skip the gopls directory in x/tools from being tested in GOPATH mode. That directory and everything inside is developed with support for module mode only. All other packages in x/tools continue to be supported both in module and GOPATH modes for the current and previous releases, as per release policy¹. ¹ https://golang.org/doc/devel/release.html#policy Fixes golang/go#34190 Change-Id: Icdb4d0e1419f69c84c9a3f9a33727a09a35599f4 Reviewed-on: https://go-review.googlesource.com/c/build/+/194397 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-09-09 23:48:08 +03:00
// Test everything else in GOPATH mode as usual.
return true
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
}
cmd/coordinator: finish the scheduler code, at least mostly Optimizations and tuning remain, but this should be tons better than what we had before (random). Updates golang/go#19178 Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de Reviewed-on: https://go-review.googlesource.com/c/build/+/205078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-30 06:59:13 +03:00
func (c *BuildConfig) IsReverse() bool { return c.HostConfig().IsReverse }
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
cmd/coordinator: finish the scheduler code, at least mostly Optimizations and tuning remain, but this should be tons better than what we had before (random). Updates golang/go#19178 Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de Reviewed-on: https://go-review.googlesource.com/c/build/+/205078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-30 06:59:13 +03:00
func (c *BuildConfig) IsContainer() bool { return c.HostConfig().IsContainer() }
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
func (c *HostConfig) IsContainer() bool { return c.ContainerImage != "" }
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
cmd/coordinator: finish the scheduler code, at least mostly Optimizations and tuning remain, but this should be tons better than what we had before (random). Updates golang/go#19178 Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de Reviewed-on: https://go-review.googlesource.com/c/build/+/205078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-30 06:59:13 +03:00
func (c *BuildConfig) IsVM() bool { return c.HostConfig().IsVM() }
dashboard: add host-linux-arm64-aws builder Add the host-linux-arm64-aws builder. The HostConfig IsVM method has been modified to support the concept that an EC2 VM can be created which runs the buildlet binary in a container. This configuration would have both a VM image and a container image set in the host configuration. For golang/go#36841 Change-Id: I092b48b1dda24d24bcd0b10dff0b2126f5f43017 Reviewed-on: https://go-review.googlesource.com/c/build/+/249118 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-18 20:32:05 +03:00
// IsVM reports whether the instance running the job is ultimately a VM. Hosts where
// a VM is used only to initiate a container are considered a container, not a VM.
// EC2 instances may be configured to run in containers that are running
// on custom AMIs.
func (c *HostConfig) IsVM() bool {
if c.isEC2 {
return c.VMImage != "" && c.ContainerImage == ""
}
return c.VMImage != ""
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
dashboard: create buildlet client package, move coordinator code into it Operation Packification, step 2 of tons. Eventually the buildlet client binary will use this stuff now. Change-Id: I4cf5f3e6beb9e56bdc795ed513ce6daaf61425e3 Reviewed-on: https://go-review.googlesource.com/2921 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 03:29:16 +03:00
func (c *BuildConfig) GOOS() string { return c.Name[:strings.Index(c.Name, "-")] }
func (c *BuildConfig) GOARCH() string {
arch := c.Name[strings.Index(c.Name, "-")+1:]
i := strings.Index(arch, "-")
if i == -1 {
return arch
}
return arch[:i]
}
cmd/coordinator: add SlowBots support, opt-in to different slow trybots This parses TRY= comments to opt-in to slower/difference trybots. This needs some docs/UI work yet. Updates golang/go#34501 Change-Id: I13a835520d7ac341bc86139b0a2118235bc83e60 Reviewed-on: https://go-review.googlesource.com/c/build/+/201338 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-16 09:02:29 +03:00
// MatchesSlowBotTerm reports whether some provided term from a
// TRY=... comment on a Run-TryBot+1 vote on Gerrit should match this
// build config.
func (c *BuildConfig) MatchesSlowBotTerm(term string) bool {
return term != "" && (term == c.Name || slowBotAliases[term] == c.Name)
}
cmd/coordinator: test sharding, better status, logs Also gomote updates which came about during the process of developing and debugging this. Change-Id: Ia53d674118a6b99bcdda7062d3b7161279b6ad52 Reviewed-on: https://go-review.googlesource.com/10463 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-28 07:51:25 +03:00
// FilePathJoin is mostly like filepath.Join (without the cleaning) except
// it uses the path separator of c.GOOS instead of the host system's.
func (c *BuildConfig) FilePathJoin(x ...string) string {
if c.GOOS() == "windows" {
return strings.Join(x, "\\")
}
return strings.Join(x, "/")
}
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
// DistTestsExecTimeout returns how long the coordinator should wait
// for a cmd/dist test execution to run the provided dist test names.
func (c *BuildConfig) DistTestsExecTimeout(distTests []string) time.Duration {
// TODO: consider using distTests? We never did before, but
// now we have the TestStats in the coordinator. Pass in a
// *buildstats.TestStats and use historical data times some
// fudge factor? For now just use the old 20 minute limit
// we've used since 2014, but scale it by the
// GO_TEST_TIMEOUT_SCALE for the super slow builders which
// struggle with, say, the cgo tests. (which should be broken
// up into separate dist tests or shards, like the test/ dir
// was)
d := 20 * time.Minute
d *= time.Duration(c.timeoutScale())
return d
}
// timeoutScale returns this builder's GO_TEST_TIMEOUT_SCALE value, or 1.
func (c *BuildConfig) timeoutScale() int {
const pfx = "GO_TEST_TIMEOUT_SCALE="
cmd/coordinator: finish the scheduler code, at least mostly Optimizations and tuning remain, but this should be tons better than what we had before (random). Updates golang/go#19178 Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de Reviewed-on: https://go-review.googlesource.com/c/build/+/205078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-30 06:59:13 +03:00
for _, env := range [][]string{c.env, c.HostConfig().env} {
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
for _, kv := range env {
if strings.HasPrefix(kv, pfx) {
if n, err := strconv.Atoi(kv[len(pfx):]); err == nil && n > 0 {
return n
}
}
}
}
return 1
}
cmd/coordinator: finish the scheduler code, at least mostly Optimizations and tuning remain, but this should be tons better than what we had before (random). Updates golang/go#19178 Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de Reviewed-on: https://go-review.googlesource.com/c/build/+/205078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-30 06:59:13 +03:00
// HostConfig returns the host configuration of c.
func (c *BuildConfig) HostConfig() *HostConfig {
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE So slow builders can finish their cgo tests, etc. Updates golang/go#29177 etc Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352 Reviewed-on: https://go-review.googlesource.com/c/162540 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-14 05:18:06 +03:00
if c.testHostConf != nil {
return c.testHostConf
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
if c, ok := Hosts[c.HostType]; ok {
return c
}
panic(fmt.Sprintf("missing buildlet config for buildlet %q", c.Name))
}
dashboard: fix docs of BuildsRepoPostSubmit, GoBootstrapURL There were some copy-paste mistakes affecting the documentation of these two identifiers. This change corrects that. Change-Id: Ic619f19c4a659a9357c816ff7fb6b0cd7b3bccd1 Reviewed-on: https://go-review.googlesource.com/c/build/+/169437 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-26 20:45:16 +03:00
// GoBootstrapURL returns the URL of a built Go 1.4+ tar.gz for the
// build configuration type c, or empty string if there isn't one.
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
func (c *BuildConfig) GoBootstrapURL(e *buildenv.Environment) string {
cmd/coordinator: finish the scheduler code, at least mostly Optimizations and tuning remain, but this should be tons better than what we had before (random). Updates golang/go#19178 Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de Reviewed-on: https://go-review.googlesource.com/c/build/+/205078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-30 06:59:13 +03:00
return strings.Replace(c.HostConfig().goBootstrapURLTmpl, "$BUCKET", e.BuildletBucket, 1)
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
}
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
// BuildletBinaryURL returns the public URL of this builder's buildlet.
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *HostConfig) BuildletBinaryURL(e *buildenv.Environment) string {
dashboard: rename Go14URL to goBootstrapURLTmpl and provide accessor Also update all users, and rename another template field. Includes changes to coordinator, gomote, and release. Change-Id: I1c4408eadbcb83d61063a910dfa18cc395952bc2 Reviewed-on: https://go-review.googlesource.com/20976 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-22 01:05:52 +03:00
tmpl := c.buildletURLTmpl
all: consolidate configuration for coordinator and gomote buildenv.Environment type defines configuration options: - Coordinator uses the GCE project name to lookup config. A custom config name can be provided at runtime to override. - The conventional prod and stage project names ('symbolic-datum-552' and 'go-dashboard-dev') map to prod and staging configuration structs. - Production and staging status is explicitly defined in configuration. - GCS bucket names for buildlet, logs, and snapshots are configurable. Change-Id: I7e6d7874eb0bdfe35dbdd5fcf6212ab50d576b88 Reviewed-on: https://go-review.googlesource.com/19502 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-15 02:59:59 +03:00
return strings.Replace(tmpl, "$BUCKET", e.BuildletBucket, 1)
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
}
cmd/coordinator: shard linux race builder Fixes golang/go#11074 Change-Id: I9401122e4d7febd3e38a80be89fcb2e03969f383 Reviewed-on: https://go-review.googlesource.com/11168 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-17 18:22:14 +03:00
func (c *BuildConfig) IsRace() bool {
return strings.HasSuffix(c.Name, "-race")
}
dashboard: use a more consistent definition for longtest builders Previously, it was possible to define a builder whose IsLongTest method would report positively, such that it'd test golang.org/x repos in long test mode, but the main Go repository in short test mode. This would be the case for any builder with "-longtest" suffix if it did not manually include GO_TEST_SHORT=0 in its environment configuration. It's unlikely we would want to do that intentionally, so this refactor makes such misconfiguration less likely by inserting the GO_TEST_SHORT environment variable assignment into the output from Env automatically. Now, a longtest builder is defined in one consistent way: it must have a "-longtest" suffix, so that the IsLongTest method reports positively. For golang/go#39054. For golang/go#29252. For golang/go#12508. Change-Id: Ic24df7b3b7de7db728bba6dc6ad4dd38a2e61e82 Reviewed-on: https://go-review.googlesource.com/c/build/+/233901 Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-13 21:30:25 +03:00
// IsLongTest reports whether this is a longtest builder.
// A longtest builder runs tests without the -short flag.
//
// A builder is considered to be a longtest builder
// if and only if its name ends with "-longtest".
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
func (c *BuildConfig) IsLongTest() bool {
return strings.HasSuffix(c.Name, "-longtest")
}
dashboard: set GOPROXY=off for the "go" repo to catch more network usage Updates golang/go#30612 Updates golang/go#30760 Change-Id: I296f4f7a163c1bdd59e839229c0961b21f26da89 Reviewed-on: https://go-review.googlesource.com/c/build/+/167039 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 07:44:17 +03:00
// OutboundNetworkAllowed reports whether this builder should be
// allowed to make outbound network requests. This is only enforced
// on some builders. (Currently most Linux ones)
func (c *BuildConfig) OutboundNetworkAllowed() bool {
dashboard: remove vetall, update misc-compile builders Updates golang/go#31916 Change-Id: I38c08955bdb4ff2b0963d5c91c6e8f78267b8004 Reviewed-on: https://go-review.googlesource.com/c/build/+/176599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-10 22:28:40 +03:00
return c.IsLongTest()
dashboard: set GOPROXY=off for the "go" repo to catch more network usage Updates golang/go#30612 Updates golang/go#30760 Change-Id: I296f4f7a163c1bdd59e839229c0961b21f26da89 Reviewed-on: https://go-review.googlesource.com/c/build/+/167039 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 07:44:17 +03:00
}
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
func (c *BuildConfig) GoInstallRacePackages() []string {
if c.InstallRacePackages != nil {
return append([]string(nil), c.InstallRacePackages...)
}
if c.IsRace() {
return []string{"std"}
}
return nil
}
dashboard: move buildlet exec code from coordinator to client package Change-Id: I778ac78ed02be9f67436ec045a3816dfc24afda3 Reviewed-on: https://go-review.googlesource.com/2923 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 20:54:03 +03:00
// AllScript returns the relative path to the operating system's script to
// do the build and run its standard set of tests.
// Example values are "src/all.bash", "src/all.bat", "src/all.rc".
func (c *BuildConfig) AllScript() string {
dashboard: add sanity check Because almost all the BuildConfig methods indirectly call AllScript, add in a little defense against accident creation of BuildConfig zero values (of the value type, not nil pointers). This can happen by doing a lookup in a map[string]dashboard.BuildConfig and then calling a method on it. We should probably move everything to pointer values of BuildConfig. There aren't many value types remaining. This paranoia isn't a result of any bug yet, but rather than the result of discussion on CL 40477. Change-Id: Ic273d4de28ac8930b56bc232bf77a9ba39f99fad Reviewed-on: https://go-review.googlesource.com/40497 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 00:37:13 +03:00
if c.Name == "" {
panic("bogus BuildConfig")
}
cmd/coordinator: shard linux race builder Fixes golang/go#11074 Change-Id: I9401122e4d7febd3e38a80be89fcb2e03969f383 Reviewed-on: https://go-review.googlesource.com/11168 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-17 18:22:14 +03:00
if c.IsRace() {
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
if strings.HasPrefix(c.Name, "windows-") {
return "src/race.bat"
}
return "src/race.bash"
}
dashboard: move buildlet exec code from coordinator to client package Change-Id: I778ac78ed02be9f67436ec045a3816dfc24afda3 Reviewed-on: https://go-review.googlesource.com/2923 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 20:54:03 +03:00
if strings.HasPrefix(c.Name, "windows-") {
return "src/all.bat"
}
if strings.HasPrefix(c.Name, "plan9-") {
return "src/all.rc"
}
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
if strings.HasPrefix(c.Name, "misc-compile") {
dashboard: add all-compile builder Runs buildall.bash only. Change-Id: I1a68d54569ccb511d956c5b181e463503353d4d4 Reviewed-on: https://go-review.googlesource.com/9512 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2015-04-30 00:44:46 +03:00
return "src/buildall.bash"
}
dashboard: move buildlet exec code from coordinator to client package Change-Id: I778ac78ed02be9f67436ec045a3816dfc24afda3 Reviewed-on: https://go-review.googlesource.com/2923 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-16 20:54:03 +03:00
return "src/all.bash"
}
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
// SplitMakeRun reports whether the coordinator should first compile
// (using c.MakeScript), then snapshot, then run the tests (ideally
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// sharded) using cmd/dist test.
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
// Eventually this function should always return true (and then be deleted)
// but for now we've only set up the scripts and verified that the main
// configurations work.
func (c *BuildConfig) SplitMakeRun() bool {
switch c.AllScript() {
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
case "src/all.bash", "src/all.bat",
"src/race.bash", "src/race.bat",
dashboard, env/linux-x86-nacl: remove nacl builders and supporting files Go 1.13 was the last release with nacl, and it's not supported by now. For golang/go#30439. Change-Id: Ic690aa388ab4505d5e755bbe40fed850a7300c69 Reviewed-on: https://go-review.googlesource.com/c/build/+/260200 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-10-07 18:12:00 +03:00
"src/all.rc":
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
// These we've verified to work.
return true
}
dashboard: remove iostest special case There are no tethered iOS builders left. If we add some in the future, they should use all.bash directly. Change-Id: Ieef7454af706c7e99c0d10fc37a337fb2c3ba130 Reviewed-on: https://go-review.googlesource.com/c/build/+/259357 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Trust: Elias Naur <mail@eliasnaur.com>
2020-10-03 12:22:43 +03:00
// TODO(bradfitz): buildall.bash should really just be N small container
// jobs instead of a "buildall.bash". Then we can delete this whole method.
dashboard, cmd/coordinator: only split make & run for builders where it'll work Change-Id: I1fdc27ea4adcc83fbd736d2bd1a34a07ca36a50c Reviewed-on: https://go-review.googlesource.com/10423 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-27 21:51:27 +03:00
return false
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
func (c *BuildConfig) IsTryOnly() bool { return c.tryOnly }
func (c *BuildConfig) NeedsGoProxy() bool { return c.needsGoProxy }
dashboard: fix docs of BuildsRepoPostSubmit, GoBootstrapURL There were some copy-paste mistakes affecting the documentation of these two identifiers. This change corrects that. Change-Id: Ic619f19c4a659a9357c816ff7fb6b0cd7b3bccd1 Reviewed-on: https://go-review.googlesource.com/c/build/+/169437 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-26 20:45:16 +03:00
// BuildsRepoPostSubmit reports whether the build configuration type c
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// should build the given repo ("go", "net", etc) and branch
// ("master", "release-branch.go1.12") as a post-submit build
// that shows up on https://build.golang.org/.
func (c *BuildConfig) BuildsRepoPostSubmit(repo, branch, goBranch string) bool {
if c.tryOnly {
dashboard: build sub-repos on more builders Change-Id: I57c12f427448281d71b8507ed950038209134143 Reviewed-on: https://go-review.googlesource.com/10915 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-11 20:01:24 +03:00
return false
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
return c.buildsRepoAtAll(repo, branch, goBranch)
cmd/coordinator: build sub-repositories Fixes golang/go#9506 Change-Id: Iebfb52386b5ba3df7d058f266e227d634d210ec6 Reviewed-on: https://go-review.googlesource.com/10732 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-05 02:45:17 +03:00
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// BuildsRepoTryBot reports whether the build configuration type c
// should build the given repo ("go", "net", etc) and branch
// ("master", "release-branch.go1.12") as a trybot.
func (c *BuildConfig) BuildsRepoTryBot(repo, branch, goBranch string) bool {
return c.tryBot != nil && c.tryBot(repo, branch, goBranch) && c.buildsRepoAtAll(repo, branch, goBranch)
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
}
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
// ShouldRunDistTest reports whether the named cmd/dist test should be
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
// run for this build config. The isNormalTry parameter is whether this
// is for a normal TryBot (non-SlowBot) run.
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
//
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
// In general, this returns true. When in normal trybot mode,
// some slow portable tests are only run on the fastest builder.
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
//
// Individual builders can adjust this policy to fit their needs.
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
func (c *BuildConfig) ShouldRunDistTest(distTest string, isNormalTry bool) bool {
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
run := true
// This section implements the default cmd/dist test policy.
// Any changes here will affect test coverage on all builders.
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
if isNormalTry {
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
slowPortableTest := distTest == "api" // Whether a test is slow and has the same behavior everywhere.
fastestBuilder := c.Name == "linux-amd64"
if slowPortableTest && !fastestBuilder {
// Don't run the test on this builder.
run = false
}
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
// Let individual builders adjust the cmd/dist test policy.
if c.distTestAdjust != nil {
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
run = c.distTestAdjust(run, distTest, isNormalTry)
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
cmd/coordinator: add a JSON mode to the /builders handler And add a test that indirectly verifies that the BuildConfig and HostConfig are JSON serializable. They weren't due to an exported func. But that exported func shouldn't be exported, so unexport it and move more policy into dashboard/builders.go. (There's been a number of recent cleanup CLs to move all policy into dashboard/builders.go instead of sprinkled all over the coordinator) A future CL will use this JSON in gomote create. Updates golang/go#30929 Change-Id: I726eaf6a4f3eeaab27d31e2642cb7642111ccd67 Reviewed-on: https://go-review.googlesource.com/c/build/+/168341 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-19 20:17:23 +03:00
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// buildsRepoAtAll reports whether we should do builds of the provided
// repo ("go", "sys", "net", etc). This applies to both post-submit
// and trybot builds. Use BuildsRepoPostSubmit for only post-submit
// or BuildsRepoTryBot for trybots.
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
//
// The branch is the branch of repo ("master",
// "release-branch.go1.12", etc); it is required. The goBranch is the
// branch of Go itself. It's required if repo != "go". When repo ==
// "go", the goBranch defaults to the value of branch.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
func (c *BuildConfig) buildsRepoAtAll(repo, branch, goBranch string) bool {
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
if goBranch == "" {
if repo == "go" {
goBranch = branch
} else {
panic("missing goBranch")
}
}
if branch == "" {
panic("missing branch")
}
if repo == "" {
panic("missing repo")
}
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
// Don't build old branches.
const minGo1x = 11
dashboard: require at least Go 1.12 for aix builder Fixes golang/go#30799 Change-Id: I0b3b53b715e2dc00f10548f74443224190003818 Reviewed-on: https://go-review.googlesource.com/c/build/+/167357 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 11:29:15 +03:00
for _, b := range []string{branch, goBranch} {
if bmaj, bmin, ok := version.ParseReleaseBranch(b); ok {
if bmaj != 1 || bmin < minGo1x {
return false
}
bmm := types.MajorMinor{bmaj, bmin}
if bmm.Less(c.MinimumGoVersion) {
return false
}
dashboard: don't test the exp repo against release branches Exp is experimental. It's hard enough keeping it passing on master. We're not going to fight release branches as well. If somebody wants to see test coverage against three versions of Go, they can graduate their code out of experimental. Change-Id: I4695ce479c47bd56a4a58dba05d367d3de020c58 Reviewed-on: https://go-review.googlesource.com/c/build/+/170518 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-03 07:45:29 +03:00
if repo == "exp" {
// Don't test exp against release branches; it's experimental.
return false
}
dashboard: require at least Go 1.12 for aix builder Fixes golang/go#30799 Change-Id: I0b3b53b715e2dc00f10548f74443224190003818 Reviewed-on: https://go-review.googlesource.com/c/build/+/167357 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 11:29:15 +03:00
}
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
}
dashboard: skip Android build on older releases Also do some cleanup of no longer applicable policy. Fixes golang/go#30485 Change-Id: Id921ea5e7f60dda097be9ebba052a09349ea000a Reviewed-on: https://go-review.googlesource.com/c/164681 Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 02:20:41 +03:00
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// Build dev.boringcrypto branches only on linux/amd64 and windows/386 (see golang.org/issue/26791).
if repo == "go" && (branch == "dev.boringcrypto" || strings.HasPrefix(branch, "dev.boringcrypto.")) {
if c.Name != "linux-amd64" && c.Name != "windows-386-2008" {
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
return false
}
}
dashboard: disable more builders Make exp off by default, as it barely passes anywhere. Lock in the places where it does pass. Also disable x/net master combined with Go 1.11 on FreeBSD 32-bit. It doesn't work on our builders and it'll never be fixed (we're not going to make changes to Go 1.11) and there's resistance to skipping those tests (my CL 170777), so I'm just turning off the builder. That's not ideal, but it's better than people getting used to red on the dashboard. Fixes golang/go#31221 Change-Id: Id0228e4b0ee73715a0c2da84f68dcfc989a48aaf Reviewed-on: https://go-review.googlesource.com/c/build/+/170782 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-05 08:26:05 +03:00
if repo != "go" && !c.SplitMakeRun() {
return false
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
if p := c.buildsRepo; p != nil {
return p(repo, branch, goBranch)
}
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
return buildRepoByDefault(repo)
dashboard: disable more builders Make exp off by default, as it barely passes anywhere. Lock in the places where it does pass. Also disable x/net master combined with Go 1.11 on FreeBSD 32-bit. It doesn't work on our builders and it'll never be fixed (we're not going to make changes to Go 1.11) and there's resistance to skipping those tests (my CL 170777), so I'm just turning off the builder. That's not ideal, but it's better than people getting used to red on the dashboard. Fixes golang/go#31221 Change-Id: Id0228e4b0ee73715a0c2da84f68dcfc989a48aaf Reviewed-on: https://go-review.googlesource.com/c/build/+/170782 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-05 08:26:05 +03:00
}
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
// buildRepoByDefault reports whether builders should do builds
// for the given repo ("go", "net", etc.) by default.
//
// It's used directly by BuildConfig.buildsRepoAtAll method for
// builders with a nil BuildConfig.buildsRepo value.
// It's also used by many builders that provide a custom build
// repo policy (a non-nil BuildConfig.buildsRepo value) as part
// of making the decision of whether to build a given repo.
//
// As a result, it effectively implements the default build repo policy.
// Any changes here will affect repo coverage of many builders.
func buildRepoByDefault(repo string) bool {
dashboard: add some "exp" trybots back, clean up some config Fixes golang/go#30781 Change-Id: I2672ca2949837c3cfe8fa6bd8011d1b9b43cca03 Reviewed-on: https://go-review.googlesource.com/c/build/+/167640 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-14 23:35:32 +03:00
switch repo {
case "go":
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
// Build the main Go repository by default.
dashboard: add some "exp" trybots back, clean up some config Fixes golang/go#30781 Change-Id: I2672ca2949837c3cfe8fa6bd8011d1b9b43cca03 Reviewed-on: https://go-review.googlesource.com/c/build/+/167640 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-14 23:35:32 +03:00
return true
dashboard: don't build x/build on so many builders It was kinda overkill. Change-Id: Ied7fc42aa59a1ed18cc1da20e51c8ed386d84629 Reviewed-on: https://go-review.googlesource.com/c/build/+/210378 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-07 08:14:28 +03:00
case "mobile", "exp", "build":
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
// Don't build x/mobile, x/exp, x/build by default.
//
// Builders need to explicitly opt-in to build these repos.
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
return false
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
default:
// Build all other golang.org/x repositories by default.
return true
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
}
cmd/coordinator: move skip branch policy into dashboard This change takes care of a TODO comment. It makes the builder configuration more centralized and contained in the dashboard package. Invert it for consistency with BuildConfig.BuildRepo method. Updates golang/go#26791 Change-Id: I46368adadb85f2ec730da4fc0abe5fd6a112a7c7 Reviewed-on: https://go-review.googlesource.com/c/149738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-15 08:55:22 +03:00
}
dashboard: disable more builders Make exp off by default, as it barely passes anywhere. Lock in the places where it does pass. Also disable x/net master combined with Go 1.11 on FreeBSD 32-bit. It doesn't work on our builders and it'll never be fixed (we're not going to make changes to Go 1.11) and there's resistance to skipping those tests (my CL 170777), so I'm just turning off the builder. That's not ideal, but it's better than people getting used to red on the dashboard. Fixes golang/go#31221 Change-Id: Id0228e4b0ee73715a0c2da84f68dcfc989a48aaf Reviewed-on: https://go-review.googlesource.com/c/build/+/170782 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-05 08:26:05 +03:00
func defaultPlusExp(repo, branch, goBranch string) bool {
if repo == "exp" {
return true
}
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
return buildRepoByDefault(repo)
dashboard: disable more builders Make exp off by default, as it barely passes anywhere. Lock in the places where it does pass. Also disable x/net master combined with Go 1.11 on FreeBSD 32-bit. It doesn't work on our builders and it'll never be fixed (we're not going to make changes to Go 1.11) and there's resistance to skipping those tests (my CL 170777), so I'm just turning off the builder. That's not ideal, but it's better than people getting used to red on the dashboard. Fixes golang/go#31221 Change-Id: Id0228e4b0ee73715a0c2da84f68dcfc989a48aaf Reviewed-on: https://go-review.googlesource.com/c/build/+/170782 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-05 08:26:05 +03:00
}
dashboard: don't build x/build on so many builders It was kinda overkill. Change-Id: Ied7fc42aa59a1ed18cc1da20e51c8ed386d84629 Reviewed-on: https://go-review.googlesource.com/c/build/+/210378 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-07 08:14:28 +03:00
func defaultPlusExpBuild(repo, branch, goBranch string) bool {
if repo == "exp" || repo == "build" {
return true
}
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
return buildRepoByDefault(repo)
dashboard: don't build x/build on so many builders It was kinda overkill. Change-Id: Ied7fc42aa59a1ed18cc1da20e51c8ed386d84629 Reviewed-on: https://go-review.googlesource.com/c/build/+/210378 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-07 08:14:28 +03:00
}
cmd/coordinator, dashboard: split build into 'make' and 'run' steps Change-Id: Ib60a601ef21163d5e0d1fed3a2609d9b719c616a Reviewed-on: https://go-review.googlesource.com/10073 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-21 07:02:38 +03:00
// AllScriptArgs returns the set of arguments that should be passed to the
dashboard: add darwin-arm/arm64 buildlets Tested by running a local coordinator on the darwin-arm64 builder. Change-Id: Id38faaa3275d6535b897f3c65df6d01ed9c1847e Reviewed-on: https://go-review.googlesource.com/9459 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-04-29 15:54:19 +03:00
// all.bash-equivalent script. Usually empty.
func (c *BuildConfig) AllScriptArgs() []string {
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
return append([]string(nil), c.allScriptArgs...)
dashboard: add darwin-arm/arm64 buildlets Tested by running a local coordinator on the darwin-arm64 builder. Change-Id: Id38faaa3275d6535b897f3c65df6d01ed9c1847e Reviewed-on: https://go-review.googlesource.com/9459 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-04-29 15:54:19 +03:00
}
cmd/release: add new program to build releases The release command will replace misc/makerelease from the main repo. The contents of makerelease make their way into this program. Also add GorootFinal and MakeScript methods to dashboard.BuildConfig. Change-Id: Id20f3f7a85a21f040d8cf47e438d299d57360556 Reviewed-on: https://go-review.googlesource.com/3693 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-02 15:05:01 +03:00
// MakeScript returns the relative path to the operating system's script to
// do the build.
// Example values are "src/make.bash", "src/make.bat", "src/make.rc".
func (c *BuildConfig) MakeScript() string {
if strings.HasPrefix(c.Name, "windows-") {
return "src/make.bat"
}
if strings.HasPrefix(c.Name, "plan9-") {
return "src/make.rc"
}
return "src/make.bash"
}
cmd/coordinator, dashboard: split build into 'make' and 'run' steps Change-Id: Ib60a601ef21163d5e0d1fed3a2609d9b719c616a Reviewed-on: https://go-review.googlesource.com/10073 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-21 07:02:38 +03:00
// MakeScriptArgs returns the set of arguments that should be passed to the
// make.bash-equivalent script. Usually empty.
func (c *BuildConfig) MakeScriptArgs() []string {
return c.AllScriptArgs()
}
cmd/release: add new program to build releases The release command will replace misc/makerelease from the main repo. The contents of makerelease make their way into this program. Also add GorootFinal and MakeScript methods to dashboard.BuildConfig. Change-Id: Id20f3f7a85a21f040d8cf47e438d299d57360556 Reviewed-on: https://go-review.googlesource.com/3693 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-02 15:05:01 +03:00
// GorootFinal returns the default install location for
// releases for this platform.
func (c *BuildConfig) GorootFinal() string {
if strings.HasPrefix(c.Name, "windows-") {
return "c:\\go"
}
return "/usr/local/go"
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
// MachineType returns the GCE machine type to use for this builder.
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *HostConfig) MachineType() string {
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
if v := c.machineType; v != "" {
return v
}
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
if c.IsContainer() {
// Set a higher default machine size for containers,
// so their /workdir tmpfs can be larger. The COS
// image has no swap, so we want to make sure the
// /workdir fits completely in memory.
dashboard: upsize machine type for windows-amd64-{longtest,race} builders Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for windows-amd64-longtest and windows-amd64-race builders. Continue to use the existing n1-highcpu-4 machine type for all other windows/386 and windows/amd64 builders for now. The primary motivation of this change is to increase amount of RAM available on the longtest builder from 3.6 GB to 7.2 GB, because it is consistenly failing due to being out of memory. Also fix minor typos, and increase consistency in comments and notes. Updates golang/go#33951 Change-Id: Iac5da67fa584059842850a9c8deda98c242741f2 Reviewed-on: https://go-review.googlesource.com/c/build/+/208500 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-11-23 02:01:25 +03:00
return "n1-standard-4" // 4 vCPUs, 15 GB mem
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
return "n1-highcpu-2"
}
dashboard, buildenv: label EC2 instances This change adds a boolean which identified when an instance is an EC2 instance. Various EC2 related variables have been renamed to be more consistent. Updates golang/go#36841 Change-Id: Ief385bf9d41e320ebd7d31bc0108e8053f337afd Reviewed-on: https://go-review.googlesource.com/c/build/+/233801 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-13 20:27:15 +03:00
// IsEC2 returns true if the machine type is an EC2 arm64 type.
func (c *HostConfig) IsEC2() bool {
return c.isEC2
}
cmd/coordinator: generate table of builders To replace the always-out-of-date wiki. Change-Id: I63d56d4bcf9252e56c833eaa15656b0321ac91dc Reviewed-on: https://go-review.googlesource.com/9571 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-01 03:03:01 +03:00
// ShortOwner returns a short human-readable owner.
func (c BuildConfig) ShortOwner() string {
cmd/coordinator: finish the scheduler code, at least mostly Optimizations and tuning remain, but this should be tons better than what we had before (random). Updates golang/go#19178 Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de Reviewed-on: https://go-review.googlesource.com/c/build/+/205078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-30 06:59:13 +03:00
owner := c.HostConfig().Owner
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
if owner == "" {
cmd/coordinator: generate table of builders To replace the always-out-of-date wiki. Change-Id: I63d56d4bcf9252e56c833eaa15656b0321ac91dc Reviewed-on: https://go-review.googlesource.com/9571 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-01 03:03:01 +03:00
return "go-dev"
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
return strings.TrimSuffix(owner, "@golang.org")
cmd/coordinator: generate table of builders To replace the always-out-of-date wiki. Change-Id: I63d56d4bcf9252e56c833eaa15656b0321ac91dc Reviewed-on: https://go-review.googlesource.com/9571 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-05-01 03:03:01 +03:00
}
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
// OwnerGithub returns the Github handle of the owner.
func (c BuildConfig) OwnerGithub() string {
cmd/coordinator: finish the scheduler code, at least mostly Optimizations and tuning remain, but this should be tons better than what we had before (random). Updates golang/go#19178 Change-Id: Idb483a4c4209a012814322cc8b37b966ee4681de Reviewed-on: https://go-review.googlesource.com/c/build/+/205078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-30 06:59:13 +03:00
return c.HostConfig().OwnerGithub
dashboard/builders: add github handles Adds an initial set of github handles for builder owners and displays a link in coordinator if set. Updates golang/go#19848 Change-Id: I85aa98a48b8a094482b22021b32b465af5313bbe Reviewed-on: https://go-review.googlesource.com/40654 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-13 22:38:46 +03:00
}
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
// PoolName returns a short summary of the builder's host type for the
all: upgrade farmer.golang.org links to https Since farmer.golang.org has a valid certificate in production now, we should use that certificate for retrieving data. Change-Id: I3685d84953ea11e6ac09b4692fa27c73538a565c Reviewed-on: https://go-review.googlesource.com/39290 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-03 17:51:28 +03:00
// https://farmer.golang.org/builders page.
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
func (c *HostConfig) PoolName() string {
switch {
case c.IsReverse:
return "Reverse (dedicated machine/VM)"
dashboard, buildenv: label EC2 instances This change adds a boolean which identified when an instance is an EC2 instance. Various EC2 related variables have been renamed to be more consistent. Updates golang/go#36841 Change-Id: Ief385bf9d41e320ebd7d31bc0108e8053f337afd Reviewed-on: https://go-review.googlesource.com/c/build/+/233801 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-13 20:27:15 +03:00
case c.IsEC2():
env/linux-arm64/aws, dashboard: update linux-arm64 builder image This updates the instruction for creating a VM image and a buildlet container image. The instructions will provide a better description for the steps required to create the images. The VM image has been updated to no longer require sudo when the docker command is executed. The host configuration has been updated with a new VM type and VM image. The rundockerbuildlet service configuration has been modified: * The memory limit has been removed because rundockerbuildlet will now use all of the available memory for buildlets running on AWS. * The service will report as running even after rundockerbuildlet has exited. Rundockerbuildlet will initiate a single buildlet and then exit. * The service will not be restarted after rundockerbuildlet has exited. Changes made to rundockerbuildlet which required these changes have been made in CL 234114. Updates golang/go#36841 Change-Id: I3ff51e414b11a2b1427cbd7c6e24e9135fc2a5c3 Reviewed-on: https://go-review.googlesource.com/c/build/+/236297 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-20 00:26:14 +03:00
return "EC2 VM Container"
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsVM():
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
return "GCE VM"
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsContainer():
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
return "Container"
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
}
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
panic("unknown builder type")
}
env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx This adds scripts to create a new builder host image that acts like Container-Optimized OS (has docker, runs konlet on startup) but with a Debian 9 kernel + userspace that permits KVM for nested virtualization. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884 Reviewed-on: https://go-review.googlesource.com/c/163301 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-22 00:10:54 +03:00
// ContainerVMImage returns the base VM name (not the fully qualified
// URL resource name of the VM) that starts the konlet program that
dashboard, buildlet, cmd/debugnewvm: clean up HostConfig.ContainerVMImage The debugnewvm command was printing out the wrong VM name, since the old method had also assumed (but not documented) that it only applied to nested virtualization containers. But remove that requirement and make the empty string mean unspecified instead (which we currently mean to use Container-Optimized OS) role and then use its new definition. Change-Id: Ieca138285aa567b1c24d585c5aa180f8a1534154 Reviewed-on: https://go-review.googlesource.com/c/build/+/177919 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-05-17 23:08:13 +03:00
// pulls & runs a container.
// The empty string means that no particular VM image is required
// and the caller can run this container in any host.
//
// This method is only applicable when c.IsContainer() is true.
env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx This adds scripts to create a new builder host image that acts like Container-Optimized OS (has docker, runs konlet on startup) but with a Debian 9 kernel + userspace that permits KVM for nested virtualization. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884 Reviewed-on: https://go-review.googlesource.com/c/163301 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-22 00:10:54 +03:00
func (c *HostConfig) ContainerVMImage() string {
if c.KonletVMImage != "" {
return c.KonletVMImage
}
dashboard, buildlet, cmd/debugnewvm: clean up HostConfig.ContainerVMImage The debugnewvm command was printing out the wrong VM name, since the old method had also assumed (but not documented) that it only applied to nested virtualization containers. But remove that requirement and make the empty string mean unspecified instead (which we currently mean to use Container-Optimized OS) role and then use its new definition. Change-Id: Ieca138285aa567b1c24d585c5aa180f8a1534154 Reviewed-on: https://go-review.googlesource.com/c/build/+/177919 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-05-17 23:08:13 +03:00
if c.NestedVirt {
return "debian-stretch-vmx"
}
env/linux-arm64/aws, dashboard: update linux-arm64 builder image This updates the instruction for creating a VM image and a buildlet container image. The instructions will provide a better description for the steps required to create the images. The VM image has been updated to no longer require sudo when the docker command is executed. The host configuration has been updated with a new VM type and VM image. The rundockerbuildlet service configuration has been modified: * The memory limit has been removed because rundockerbuildlet will now use all of the available memory for buildlets running on AWS. * The service will report as running even after rundockerbuildlet has exited. Rundockerbuildlet will initiate a single buildlet and then exit. * The service will not be restarted after rundockerbuildlet has exited. Changes made to rundockerbuildlet which required these changes have been made in CL 234114. Updates golang/go#36841 Change-Id: I3ff51e414b11a2b1427cbd7c6e24e9135fc2a5c3 Reviewed-on: https://go-review.googlesource.com/c/build/+/236297 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-20 00:26:14 +03:00
if c.isEC2 && c.ContainerImage != "" {
return fmt.Sprintf("gcr.io/%s/%s", buildenv.Production.ProjectName, c.ContainerImage)
}
dashboard, buildlet, cmd/debugnewvm: clean up HostConfig.ContainerVMImage The debugnewvm command was printing out the wrong VM name, since the old method had also assumed (but not documented) that it only applied to nested virtualization containers. But remove that requirement and make the empty string mean unspecified instead (which we currently mean to use Container-Optimized OS) role and then use its new definition. Change-Id: Ieca138285aa567b1c24d585c5aa180f8a1534154 Reviewed-on: https://go-review.googlesource.com/c/build/+/177919 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-05-17 23:08:13 +03:00
return ""
env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx This adds scripts to create a new builder host image that acts like Container-Optimized OS (has docker, runs konlet on startup) but with a Debian 9 kernel + userspace that permits KVM for nested virtualization. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884 Reviewed-on: https://go-review.googlesource.com/c/163301 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-22 00:10:54 +03:00
}
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
// IsHermetic reports whether this host config gets a fresh
// environment (including /usr, /var, etc) for each execution. This is
// true for VMs, GKE, and reverse buildlets running their containers
// running in Docker, but false on some reverse buildlets.
func (c *HostConfig) IsHermetic() bool {
switch {
case c.IsReverse:
return c.HermeticReverse
dashboard, buildenv: label EC2 instances This change adds a boolean which identified when an instance is an EC2 instance. Various EC2 related variables have been renamed to be more consistent. Updates golang/go#36841 Change-Id: Ief385bf9d41e320ebd7d31bc0108e8053f337afd Reviewed-on: https://go-review.googlesource.com/c/build/+/233801 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-13 20:27:15 +03:00
case c.IsEC2():
return true
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsVM():
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
return true
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
case c.IsContainer():
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
return true
}
panic("unknown builder type")
cmd/coordinator: fix the /builders page, add Host info Change-Id: I8828d62918e29c732592d82f49c48a8c3442c7f3 Reviewed-on: https://go-review.googlesource.com/29674 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 03:06:28 +03:00
}
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
// GCENumCPU reports the number of GCE CPUs this buildlet requires.
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
func (c *HostConfig) GCENumCPU() int {
buildlet, cmd/coordinator: GCE quota accounting, fixes And deal with Preemptible resource exhaustion errors. And change all-compile to misc-compile and only do the builders not covered otherwise (Fixes #11073) And make the watcher serve git source. And cache and singleflight fetching of git source. And a million other things. Fixes golang/go#11073 Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc Reviewed-on: https://go-review.googlesource.com/10750 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-06-05 04:25:50 +03:00
t := c.MachineType()
n, _ := strconv.Atoi(t[strings.LastIndex(t, "-")+1:])
return n
}
dashboard: try to speed up pre-submit longtest builders The longtest builders are currently primarily post-submit builders, where it's okay for them to be as slow as they need to be in order to provide additional test coverage. In this context, whether they take 40 minutes or 50 makes little difference. The longtest builders are also sometimes requested via SlowBots for changes that are riskier than usual, or otherwise desire additional coverage beyond the normal TryBots. They're also always enabled for CLs to release branches. In such contexts, speeding up SlowBot runs from 40 minutes to 20 or less would be appreciated and in turn help people use longtest SlowBots more frequently. Longtest builders are already configured to use sharded tests. Configure them to use additional helpers to speed up test execution. Try out 3, 5, and 9 helpers to see how much it helps before settling. For golang/go#37439. Change-Id: I425bc0257b7a54bb32c0eb1719fea7ba3f4fd461 Reviewed-on: https://go-review.googlesource.com/c/build/+/268037 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-11-06 15:55:36 +03:00
// NumTestHelpers reports how many additional buildlets
// past the first one to help out with sharded tests.
//
// isTry specifies whether it's for a pre-submit test
// run (a TryBot or SlowBot) where speed matters more.
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
func (c *BuildConfig) NumTestHelpers(isTry bool) int {
if isTry && c.numTryTestHelpers != 0 {
return c.numTryTestHelpers
}
return c.numTestHelpers
}
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
// defaultTrySet returns a trybot policy function that reports whether
// a project should use trybots. All the default projects are included,
// plus any given in extraProj.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
func defaultTrySet(extraProj ...string) func(proj, branch, goBranch string) bool {
return func(proj, branch, goBranch string) bool {
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
if proj == "go" {
return true
}
for _, p := range extraProj {
if proj == p {
return true
}
}
switch proj {
dashboard: add some "exp" trybots back, clean up some config Fixes golang/go#30781 Change-Id: I2672ca2949837c3cfe8fa6bd8011d1b9b43cca03 Reviewed-on: https://go-review.googlesource.com/c/build/+/167640 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-14 23:35:32 +03:00
case "grpc-review":
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
return false
}
return true
}
}
// explicitTrySet returns a trybot policy function that reports
// whether a project should use trybots. Only the provided projects in
// projs are enabled.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
func explicitTrySet(projs ...string) func(proj, branch, goBranch string) bool {
return func(proj, branch, goBranch string) bool {
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
for _, p := range projs {
if proj == p {
return true
}
}
return false
}
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
func init() {
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
addBuilder(BuildConfig{
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
Name: "freebsd-amd64-11_1",
HostType: "host-freebsd-11_1",
tryBot: nil,
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: re-enable FreeBSD 11.1 builders on Go 1.14 and 1.13 These builders are still used for making minor Go 1.14 and 1.13 releases at this time (see golang.org/issue/40563). Re-enable them on relevant release branches so that we are better informed for future minor releases. For golang/go#40563. Updates golang/go#40562. Change-Id: I20a6becbdb1078b92dbf6042edca7f482bca6432 Reviewed-on: https://go-review.googlesource.com/c/build/+/246638 Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-08-04 18:53:43 +03:00
// This builder is unfortunately still used by Go 1.14 and 1.13,
// so keep it around a bit longer. See golang.org/issue/40563.
// Test relevant Go versions so that we're better informed.
return atMostGo1(goBranch, 14) && buildRepoByDefault(repo)
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
},
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
distTestAdjust: fasterTrybots,
dashboard: update Windows builders, remove old manual Windows builder And clean up some other builders. Change-Id: I1f71836da7008e58d3e76e2cc3170e96cd57ddf6 Reviewed-on: https://go-review.googlesource.com/51432 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-27 07:52:12 +03:00
numTryTestHelpers: 4,
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
})
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
addBuilder(BuildConfig{
Name: "freebsd-amd64-11_2",
HostType: "host-freebsd-11_2",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("sys"),
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
distTestAdjust: fasterTrybots,
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
numTryTestHelpers: 4,
})
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
addBuilder(BuildConfig{
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
Name: "freebsd-amd64-12_0",
HostType: "host-freebsd-12_0",
MinimumGoVersion: types.MajorMinor{1, 11},
tryBot: defaultTrySet("sys"),
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
distTestAdjust: fasterTrybots, // If changing this policy, update TestShouldRunDistTest accordingly.
dashboard: add FreeBSD 12.0-alpha8 builder Change-Id: I4f0278e41d801ae0f4358d73fcc2e17456091af6 Reviewed-on: https://go-review.googlesource.com/c/140179 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-05 23:57:29 +03:00
numTryTestHelpers: 4,
})
dashboard: add freebsd-386-12_0 Fixes golang/go#30787 Change-Id: I33d892fe56abcb797c76541fb3b3a12ffe69c598 Reviewed-on: https://go-review.googlesource.com/c/build/+/167199 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-13 00:13:50 +03:00
addBuilder(BuildConfig{
Name: "freebsd-386-12_0",
HostType: "host-freebsd-12_0",
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
distTestAdjust: fasterTrybots,
dashboard: add freebsd-386-12_0 Fixes golang/go#30787 Change-Id: I33d892fe56abcb797c76541fb3b3a12ffe69c598 Reviewed-on: https://go-review.googlesource.com/c/build/+/167199 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-13 00:13:50 +03:00
numTryTestHelpers: 4,
})
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "freebsd-amd64-race",
dashboard: re-enable FreeBSD 11.1 builders on Go 1.14 and 1.13 These builders are still used for making minor Go 1.14 and 1.13 releases at this time (see golang.org/issue/40563). Re-enable them on relevant release branches so that we are better informed for future minor releases. For golang/go#40563. Updates golang/go#40562. Change-Id: I20a6becbdb1078b92dbf6042edca7f482bca6432 Reviewed-on: https://go-review.googlesource.com/c/build/+/246638 Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-08-04 18:53:43 +03:00
HostType: "host-freebsd-11_1-big", // TODO(golang.org/issue/40562): Update to newer FreeBSD.
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
})
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "freebsd-386-11_1",
HostType: "host-freebsd-11_1",
distTestAdjust: noTestDirAndNoReboot,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: re-enable FreeBSD 11.1 builders on Go 1.14 and 1.13 These builders are still used for making minor Go 1.14 and 1.13 releases at this time (see golang.org/issue/40563). Re-enable them on relevant release branches so that we are better informed for future minor releases. For golang/go#40563. Updates golang/go#40562. Change-Id: I20a6becbdb1078b92dbf6042edca7f482bca6432 Reviewed-on: https://go-review.googlesource.com/c/build/+/246638 Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-08-04 18:53:43 +03:00
// This builder is unfortunately still used by Go 1.14 and 1.13,
// so keep it around a bit longer. See golang.org/issue/40563.
// Test relevant Go versions so that we're better informed.
return atMostGo1(goBranch, 14) && buildRepoByDefault(repo)
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
},
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
dashboard: add FreeBSD 11 builders Fixes golang/go#19097 Change-Id: I29725ef511f03e10d93561889b1a4a00eb959177 Reviewed-on: https://go-review.googlesource.com/37498 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-27 19:05:25 +03:00
})
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "freebsd-386-11_2",
HostType: "host-freebsd-11_2",
distTestAdjust: noTestDirAndNoReboot,
tryBot: explicitTrySet("sys"),
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
dashboard: add FreeBSD 11.2 and 10.4 builders Not yet the defaults or trybots until we see they're working regularly for a few days and compare timings, etc. Updates golang/go#25289 Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0 Reviewed-on: https://go-review.googlesource.com/138258 Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-09-28 02:43:25 +03:00
})
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "linux-386",
HostType: "host-linux-jessie",
distTestAdjust: fasterTrybots,
tryBot: defaultTrySet(),
env, buildlet, dashboard: support outbound firewalls for Linux Updates golang/go#30612 Change-Id: Ib13a286d0944a7f4a13b9e93a01533693052858b Reviewed-on: https://go-review.googlesource.com/c/build/+/165637 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 05:06:17 +03:00
env: []string{
"GOARCH=386",
"GOHOSTARCH=386",
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
env, buildlet, dashboard: support outbound firewalls for Linux Updates golang/go#30612 Change-Id: Ib13a286d0944a7f4a13b9e93a01533693052858b Reviewed-on: https://go-review.googlesource.com/c/build/+/165637 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 05:06:17 +03:00
},
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
numTestHelpers: 1,
numTryTestHelpers: 3,
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
dashboard: add linux-386-softfloat builder It's based on the linux-386-387 builder, but with GO386=softfloat and Debian 9 (stretch) instead of Debian 8 (jessie). Test it on a subset of golang.org/x repos (golang.org/x/crypto), since that should provide sufficient test coverage. For golang/go#41840. For golang/go#41848. Change-Id: Ida95fce198056a05c359ac5ab816c582bdd5930f Reviewed-on: https://go-review.googlesource.com/c/build/+/260201 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-10-07 18:24:42 +03:00
addBuilder(BuildConfig{
Name: "linux-386-softfloat",
Notes: "GO386=softfloat",
buildsRepo: func(repo, branch, goBranch string) bool {
// GO386=softfloat is added in Go 1.16 (golang.org/issue/41848).
return atLeastGo1(goBranch, 16) && (repo == "go" || repo == "crypto")
},
GoDeps: []string{
"04b8a9fea57e37589d82410281f22ebde0027808", // CL 260017, "all: implement GO386=softfloat".
},
HostType: "host-linux-stretch",
env: []string{"GOARCH=386", "GOHOSTARCH=386", "GO386=softfloat"},
})
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
addBuilder(BuildConfig{
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
Name: "linux-386-387",
Notes: "GO386=387",
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: only run linux-386-387 builder on Go 1.15 and older Also update some test cases to be more relevant. For golang/go#40255. Fixes golang/go#41799. Change-Id: Ie63e068a46058bcaf7866f2090049fb888612cbe Reviewed-on: https://go-review.googlesource.com/c/build/+/259449 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Trust: Dmitri Shuralyov <dmitshur@golang.org>
2020-10-05 19:20:33 +03:00
// GO386=387 is removed in Go 1.16 (golang.org/issue/40255).
// It's still supported in Go 1.15 and 1.14.
return atMostGo1(goBranch, 15) && (repo == "go" || repo == "crypto")
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
},
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386", "GO386=387"},
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
addBuilder(BuildConfig{
dashboard: disable more builders Make exp off by default, as it barely passes anywhere. Lock in the places where it does pass. Also disable x/net master combined with Go 1.11 on FreeBSD 32-bit. It doesn't work on our builders and it'll never be fixed (we're not going to make changes to Go 1.11) and there's resistance to skipping those tests (my CL 170777), so I'm just turning off the builder. That's not ideal, but it's better than people getting used to red on the dashboard. Fixes golang/go#31221 Change-Id: Id0228e4b0ee73715a0c2da84f68dcfc989a48aaf Reviewed-on: https://go-review.googlesource.com/c/build/+/170782 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-05 08:26:05 +03:00
Name: "linux-amd64",
HostType: "host-linux-stretch",
tryBot: defaultTrySet(),
dashboard: don't build x/build on so many builders It was kinda overkill. Change-Id: Ied7fc42aa59a1ed18cc1da20e51c8ed386d84629 Reviewed-on: https://go-review.googlesource.com/c/build/+/210378 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-07 08:14:28 +03:00
buildsRepo: defaultPlusExpBuild,
env, buildlet, dashboard: support outbound firewalls for Linux Updates golang/go#30612 Change-Id: Ib13a286d0944a7f4a13b9e93a01533693052858b Reviewed-on: https://go-review.googlesource.com/c/build/+/165637 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 05:06:17 +03:00
env: []string{
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
env, buildlet, dashboard: support outbound firewalls for Linux Updates golang/go#30612 Change-Id: Ib13a286d0944a7f4a13b9e93a01533693052858b Reviewed-on: https://go-review.googlesource.com/c/build/+/165637 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 05:06:17 +03:00
},
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
numTestHelpers: 1,
numTryTestHelpers: 4,
RunBench: true,
all: run dev coordinator locally, allow debug in prod * flag allows debug (trigger a build) in prod mode * use default zone (us-central1-f) in staging * compute.CloudPlatformScope satisfies compute and storage scope requirements * new buildlet for linux-amd64 on Kubernetes Updates golang/go#12546 Change-Id: I1a0cbc0e0c0552b7ac56943c646378194508d48f Reviewed-on: https://go-review.googlesource.com/17102 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-11-20 21:00:06 +03:00
})
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
addBuilder(BuildConfig{
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
Name: "linux-amd64-vmx",
HostType: "host-linux-stretch-vmx",
buildsRepo: disabledBuilder,
dashboard, buildlet: add a disabled builder with nested virt, for testing This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-16 01:05:42 +03:00
})
dashboard: remove alpine builder It's doing no good being red all the time, especially as nobody is working on it. Updates golang/go#22689 Updates golang/go#19938 Change-Id: Icb947878d85e920f24ea458eb0f319844ca5bd60 Reviewed-on: https://go-review.googlesource.com/78575 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-18 06:31:35 +03:00
const testAlpine = false // Issue 22689 (hide all red builders), Issue 19938 (get Alpine passing)
if testAlpine {
addBuilder(BuildConfig{
Name: "linux-amd64-alpine",
HostType: "host-linux-x86-alpine",
})
}
dashboard: promote misc-compile-darwinarm64 builder to a TryBot Testing shows that misc-compile-darwinarm64 is working as expected on recent commits. Promote it to a normal misc-compile TryBot now, with the exception that it needs to run on Go 1.16 and newer only. Fixes golang/go#42341. Change-Id: Ib4de19a2ff904fb9de056db5076ac63690eb934c Reviewed-on: https://go-review.googlesource.com/c/build/+/267123 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-11-03 06:25:51 +03:00
// addMiscCompileGo1 adds a misc-compile TryBot that runs
dashboard: remove vetall, update misc-compile builders Updates golang/go#31916 Change-Id: I38c08955bdb4ff2b0963d5c91c6e8f78267b8004 Reviewed-on: https://go-review.googlesource.com/c/build/+/176599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-10 22:28:40 +03:00
// buildall.bash on a subset of platforms matching the egrep
// pattern rx. The pattern is matched against the "go tool
// dist list" name, but with hyphens instead of forward
// slashes ("linux-amd64", etc).
dashboard: promote misc-compile-darwinarm64 builder to a TryBot Testing shows that misc-compile-darwinarm64 is working as expected on recent commits. Promote it to a normal misc-compile TryBot now, with the exception that it needs to run on Go 1.16 and newer only. Fixes golang/go#42341. Change-Id: Ib4de19a2ff904fb9de056db5076ac63690eb934c Reviewed-on: https://go-review.googlesource.com/c/build/+/267123 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-11-03 06:25:51 +03:00
// If min is non-zero, it specifies the minimum Go 1.x version.
addMiscCompileGo1 := func(min int, suffix, rx string) {
var v types.MajorMinor
var alsoNote string
if min != 0 {
v = types.MajorMinor{1, min}
alsoNote = fmt.Sprintf(" Applies to Go 1.%d and newer.", min)
}
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
addBuilder(BuildConfig{
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
Name: "misc-compile" + suffix,
HostType: "host-linux-jessie",
tryBot: defaultTrySet(),
env: []string{
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
},
dashboard: promote misc-compile-darwinarm64 builder to a TryBot Testing shows that misc-compile-darwinarm64 is working as expected on recent commits. Promote it to a normal misc-compile TryBot now, with the exception that it needs to run on Go 1.16 and newer only. Fixes golang/go#42341. Change-Id: Ib4de19a2ff904fb9de056db5076ac63690eb934c Reviewed-on: https://go-review.googlesource.com/c/build/+/267123 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-11-03 06:25:51 +03:00
tryOnly: true,
MinimumGoVersion: v,
CompileOnly: true,
Notes: "Runs buildall.bash to cross-compile & vet std+cmd packages for " + rx + ", but doesn't run any tests." + alsoNote,
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
allScriptArgs: []string{
// Filtering pattern to buildall.bash:
rx,
},
})
}
dashboard: promote misc-compile-darwinarm64 builder to a TryBot Testing shows that misc-compile-darwinarm64 is working as expected on recent commits. Promote it to a normal misc-compile TryBot now, with the exception that it needs to run on Go 1.16 and newer only. Fixes golang/go#42341. Change-Id: Ib4de19a2ff904fb9de056db5076ac63690eb934c Reviewed-on: https://go-review.googlesource.com/c/build/+/267123 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-11-03 06:25:51 +03:00
// addMiscCompile adds a misc-compile TryBot
// for all supported Go versions.
addMiscCompile := func(suffix, rx string) { addMiscCompileGo1(0, suffix, rx) }
dashboard: create misc-compile-darwinarm64 builder Create the builder as a post-submit builder with a non-zero KnownIssue value first. Once we know it works well, we'll have the confidence to be able to safely promote it to a normal TryBot, without the risk of causing disruption for contributors. It's different from misc-compile-darwin since it needs to run on recent Go 1.16 commits only. For golang/go#42341. Change-Id: I67170988d7323cb73439755a3961532c668040e7 Reviewed-on: https://go-review.googlesource.com/c/build/+/267105 Trust: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-11-02 20:41:26 +03:00
dashboard: verify that each port is covered by real or misc-compile trybot Currently, android/{386,arm,arm64} are not covered, so add a misc-compile builder for these. Also correct a typo in the comment for the darwin misc-compile builder and add update the misc-compile-freebsd comment to include freebsd/arm64. Change-Id: Ica3b27e98db73c10edf30260acddcd7ea57d9051 Reviewed-on: https://go-review.googlesource.com/c/build/+/205798 Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2019-11-07 16:15:55 +03:00
addMiscCompile("-linuxarm", "^linux-arm") // 2: arm, arm64
dashboard: remove misc-compile-ios builder Since sending CL 257617, I've learned that iOS was never a supported target for misc-compile. It was always filtered out unconditionally in GOROOT/src/buildall.bash, and the comment suggesting that we test iOS targets was inaccurate. Remove the naive misc-compile-ios builder since it's not expected to start working soon, and update misc-builder architecture filtering code and comments to match the current reality better. After this change, it will be viable to remove mobile target filtering from GOROOT/src/buildall.bash. If that's done, then x/build/dashboard will become a centralized place for all misc-builder target selection. Ignore darwin-arm in TestTryBotsCompileAllPorts so the test doesn't fail when executed with Go 1.14. Not having that builder is intended. For golang/go#41610. Updates golang/go#25963. Change-Id: Ic6f5dc95edb67f8e6ebc0a77979e55eb0cc325cb Reviewed-on: https://go-review.googlesource.com/c/build/+/258057 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-09-28 18:04:35 +03:00
addMiscCompile("-darwin", "^darwin-(386|amd64)$") // 1: amd64 (in Go 1.14: 386, amd64)
dashboard: promote misc-compile-darwinarm64 builder to a TryBot Testing shows that misc-compile-darwinarm64 is working as expected on recent commits. Promote it to a normal misc-compile TryBot now, with the exception that it needs to run on Go 1.16 and newer only. Fixes golang/go#42341. Change-Id: Ib4de19a2ff904fb9de056db5076ac63690eb934c Reviewed-on: https://go-review.googlesource.com/c/build/+/267123 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-11-03 06:25:51 +03:00
addMiscCompileGo1(16, "-darwinarm64", "^darwin-arm64$") // 1: arm64 (for Go 1.16 and newer)
dashboard: verify that each port is covered by real or misc-compile trybot Currently, android/{386,arm,arm64} are not covered, so add a misc-compile builder for these. Also correct a typo in the comment for the darwin misc-compile builder and add update the misc-compile-freebsd comment to include freebsd/arm64. Change-Id: Ica3b27e98db73c10edf30260acddcd7ea57d9051 Reviewed-on: https://go-review.googlesource.com/c/build/+/205798 Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2019-11-07 16:15:55 +03:00
addMiscCompile("-mips", "^linux-mips") // 4: mips, mipsle, mips64, mips64le
addMiscCompile("-ppc", "^(linux-ppc64|aix-)") // 3: linux-ppc64{,le}, aix-ppc64
addMiscCompile("-solaris", "^(solaris|illumos)") // 2: both amd64
addMiscCompile("-plan9", "^plan9-") // 3: amd64, 386, arm
addMiscCompile("-freebsd", `^freebsd-(386|arm|arm64)\b`) // 3: 386, arm, arm64 (amd64 already trybot)
addMiscCompile("-netbsd", "^netbsd-") // 4: amd64, 386, arm, arm64
addMiscCompile("-openbsd", "^openbsd-") // 4: amd64, 386, arm, arm64
dashboard: remove misc-compile-android builder Fixes golang/go#35596 Updates golang/go#25963 Change-Id: I9db5154dcdd0e432dfe2d0d8697fedabed50a7ed Reviewed-on: https://go-review.googlesource.com/c/build/+/207377 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-15 07:29:29 +03:00
dashboard: remove the Windows ARM builder The host-windows-arm-iotcore builder has been missing for an extended period of time. Attempts to contact the maintainer have failed. This removes the builder until we can find a replacement for windows-arm instances. Fixes golang/go#38493 For golang/go#38607 Change-Id: Ia7d2168f676a86bced754688b4e2d52e61f3125b Reviewed-on: https://go-review.googlesource.com/c/build/+/229559 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-04-23 01:52:59 +03:00
// And 3 that don't fit above:
addMiscCompile("-other", "^(linux-s390x|linux-riscv64|dragonfly-amd64)$")
dashboard: remove misc-compile-ios builder Since sending CL 257617, I've learned that iOS was never a supported target for misc-compile. It was always filtered out unconditionally in GOROOT/src/buildall.bash, and the comment suggesting that we test iOS targets was inaccurate. Remove the naive misc-compile-ios builder since it's not expected to start working soon, and update misc-builder architecture filtering code and comments to match the current reality better. After this change, it will be viable to remove mobile target filtering from GOROOT/src/buildall.bash. If that's done, then x/build/dashboard will become a centralized place for all misc-builder target selection. Ignore darwin-arm in TestTryBotsCompileAllPorts so the test doesn't fail when executed with Go 1.14. Not having that builder is intended. For golang/go#41610. Updates golang/go#25963. Change-Id: Ic6f5dc95edb67f8e6ebc0a77979e55eb0cc325cb Reviewed-on: https://go-review.googlesource.com/c/build/+/258057 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Trust: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-09-28 18:04:35 +03:00
// TODO: Issue 25963, get the misc-compile trybots for Android/iOS.
// Then consider subrepos too, so "mobile" can at least be included
// as a misc-compile for ^android- and ^ios-.
cmd/coordinator: shard the misc-compile builder over 4 builders Now that it's on Kubernetes, it's fast and cheap to shard it wider. Change-Id: I2858f4c417ef26661575a348fbfc47f4c25d81a2 Reviewed-on: https://go-review.googlesource.com/22795 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-05-05 03:42:49 +03:00
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "linux-amd64-nocgo",
HostType: "host-linux-jessie",
Notes: "cgo disabled",
dashboard: add some "exp" trybots back, clean up some config Fixes golang/go#30781 Change-Id: I2672ca2949837c3cfe8fa6bd8011d1b9b43cca03 Reviewed-on: https://go-review.googlesource.com/c/build/+/167640 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-14 23:35:32 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
switch repo {
case "perf":
// Requires sqlite, which requires cgo.
return false
case "mobile":
return false
dashboard: stop testing x/build on even more builders Follow up to CL 210378 after looking at: https://build.golang.org/?repo=golang.org%2fx%2fbuild Updates golang/go#30640 Change-Id: I81505a9ca2985ffa6e75922665b14ef4de2af860 Reviewed-on: https://go-review.googlesource.com/c/build/+/211100 Reviewed-by: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-12-12 20:43:59 +03:00
case "build":
return false
dashboard: add some "exp" trybots back, clean up some config Fixes golang/go#30781 Change-Id: I2672ca2949837c3cfe8fa6bd8011d1b9b43cca03 Reviewed-on: https://go-review.googlesource.com/c/build/+/167640 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-14 23:35:32 +03:00
}
return true
},
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
env: []string{
"CGO_ENABLED=0",
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
// This USER=root was required for Docker-based builds but probably isn't required
// in the VM anymore, since the buildlet probably already has this in its environment.
// (It was required because without cgo, it couldn't find the username)
"USER=root",
},
})
addBuilder(BuildConfig{
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
Name: "linux-amd64-noopt",
Notes: "optimizations and inlining disabled",
HostType: "host-linux-jessie",
buildsRepo: onlyGo,
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
"GO_GCFLAGS=-N -l",
},
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
})
dashboard: add SSA check builder Now that SSA checks are off by default, let's run a builder with the checks on just to make sure they keep passing. Change-Id: I7edd2a2865681a135ccf755be7a14f8645c0a8aa Reviewed-on: https://go-review.googlesource.com/20882 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-19 01:30:38 +03:00
addBuilder(BuildConfig{
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
Name: "linux-amd64-ssacheck",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
buildsRepo: onlyGo,
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil, // TODO: add a func to conditionally run this trybot if compiler dirs are touched
build: move Linux and nacl builders to Kubernetes, add ssacheck trybot And shard the nacl builders. Fixes golang/go#15242 Fixes golang/go#12546 Change-Id: I3311678dbb64a76026f75f5691cffac0f25d976b Reviewed-on: https://go-review.googlesource.com/22780 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Ross Light <light@google.com>
2016-05-04 20:47:12 +03:00
CompileOnly: true,
Notes: "SSA internal checks enabled",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
"GO_GCFLAGS=-d=ssa/check/on,dclstack",
},
cmd/coordinator: move git ancestor checking before buildlet creation, generalize This also adds GoDeps to the builders config file and adds a dependency for the ssacheck builder, which currently fails on older release branches with: > Failed on linux-amd64-ssacheck: > https://storage.googleapis.com/go-build-log/1255edaf/linux-amd64-ssacheck_18f7e015.log > > ... > compile: unknown debug key -d dclstack Change-Id: I4ad0a298705feed56f467f3c8d9488f8be2e4af5 Reviewed-on: https://go-review.googlesource.com/43774 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-20 00:23:14 +03:00
GoDeps: []string{
"f65abf6ddc8d1f3d403a9195fd74eaffa022b07f", // adds dclstack
},
dashboard: add SSA check builder Now that SSA checks are off by default, let's run a builder with the checks on just to make sure they keep passing. Change-Id: I7edd2a2865681a135ccf755be7a14f8645c0a8aa Reviewed-on: https://go-review.googlesource.com/20882 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-03-19 01:30:38 +03:00
})
dashboard: add linux-amd64-staticlockranking builder This is a new linux/amd64 GCE builder for the purpose of testing the GOEXPERIMENT=staticlockranking configuration which is landing to master branch soon. It runs only on the main Go repository master branch and release branches 1.15 and newer. No golang.org/x repos. Not included in trybots by default, but can be requested via SlowBots by saying TRY=linux-amd64-staticlockranking. Also expand documentation of BuildConfig.buildsRepo field to make it more clear how to use it. A common mistake for custom buildsRepo functions that are added is to create a custom policy with a minor adjustment and forgetting to preserve the rest of the default build repo policy. Fixes golang/go#37937 Change-Id: Ic56f82e9ede30cb6fe8f6cc7d0282d6a0a12c1b8 Reviewed-on: https://go-review.googlesource.com/c/build/+/224078 Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-03-19 17:34:11 +03:00
addBuilder(BuildConfig{
dashboard: unset known issue for staticlockranking builder CL 207619 has been submitted to master, fixing golang.org/issue/38029. The linux-amd64-staticlockranking builder has been passing since then. Unset its known issue so that if there are regressions in the future, they can be investigated. For golang/go#38029. For golang/go#37937. Change-Id: I0de8e4ac122effee56aac73c741fab41512ff0c2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227544 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dan Scales <danscales@google.com>
2020-04-08 19:13:38 +03:00
Name: "linux-amd64-staticlockranking",
HostType: "host-linux-stretch",
Notes: "builder with GOEXPERIMENT=staticlockranking, see golang.org/issue/37937",
dashboard: add linux-amd64-staticlockranking builder This is a new linux/amd64 GCE builder for the purpose of testing the GOEXPERIMENT=staticlockranking configuration which is landing to master branch soon. It runs only on the main Go repository master branch and release branches 1.15 and newer. No golang.org/x repos. Not included in trybots by default, but can be requested via SlowBots by saying TRY=linux-amd64-staticlockranking. Also expand documentation of BuildConfig.buildsRepo field to make it more clear how to use it. A common mistake for custom buildsRepo functions that are added is to create a custom policy with a minor adjustment and forgetting to preserve the rest of the default build repo policy. Fixes golang/go#37937 Change-Id: Ic56f82e9ede30cb6fe8f6cc7d0282d6a0a12c1b8 Reviewed-on: https://go-review.googlesource.com/c/build/+/224078 Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-03-19 17:34:11 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
return repo == "go" && atLeastGo1(goBranch, 15)
},
env: []string{
"GO_DISABLE_OUTBOUND_NETWORK=1",
"GOEXPERIMENT=staticlockranking",
},
GoDeps: []string{
"02057906f7272a4787b8a0b5b7cafff8ad3024f0", // A master commit from 2020/03/19, just before CL 222925 and CL 207619 have landed.
},
})
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-racecompile",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: nil, // TODO: add a func to conditionally run this trybot if compiler dirs are touched
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
CompileOnly: true,
cmd/coordinator: make linux-amd64-racecompile rebuild std/cmd with -c=128 Updates golang/go#19962 Change-Id: I8b938a39b33d4e6b9b8dc5bc0802da95e889ee46 Reviewed-on: https://go-review.googlesource.com/41613 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-25 00:14:05 +03:00
SkipSnapshot: true,
StopAfterMake: true,
dashboard: use race-enabled linker in racecompile builder When the compiler's backend became concurrent, we introduced the racecompile builder to run the compiler with the race detector enabled to catch races. In the new linker work the linker will become more concurrent as well. It is probably a good idea to run the linker with the race detector as well. Change-Id: Ib50f030cba0509730714b74462f5a8942fe2fd2d Reviewed-on: https://go-review.googlesource.com/c/build/+/226369 Run-TryBot: Cherry Zhang <cherryyz@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-03-30 18:20:59 +03:00
InstallRacePackages: []string{"cmd/compile", "cmd/link"},
Notes: "race-enabled cmd/compile and cmd/link",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
},
cmd/coordinator: move git ancestor checking before buildlet creation, generalize This also adds GoDeps to the builders config file and adds a dependency for the ssacheck builder, which currently fails on older release branches with: > Failed on linux-amd64-ssacheck: > https://storage.googleapis.com/go-build-log/1255edaf/linux-amd64-ssacheck_18f7e015.log > > ... > compile: unknown debug key -d dclstack Change-Id: I4ad0a298705feed56f467f3c8d9488f8be2e4af5 Reviewed-on: https://go-review.googlesource.com/43774 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-20 00:23:14 +03:00
GoDeps: []string{
"22f1b56dab29d397d2bdbdd603d85e60fb678089", // adds cmd/compile -c; Issue 20222
},
dashboard, cmd/coordinator: add -racecompile builder Change-Id: Idc5756f82081de6b5dfd17036f2bfc9cf11c5659 Reviewed-on: https://go-review.googlesource.com/41414 Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-04-22 02:41:12 +03:00
})
build: convert most Linux builders to be VM-based We still use Dockerfiles to describe them, but then a new tool (docker2boot) converts them into VM images suitable for booting on GCE, running the buildlet like all the other operating systems. Things are easier if everything acts the same way. Note that since we're no longer moving around Docker images, the image size and the layer accumulation cruft no longer matters. We can now have Dockerfile lines like "RUN rm -rf /usr/share/doc" and it actually results in a smaller VM image, since we just "docker export" the files out of it to create the VM image. This doesn't yet convert the clang, sid, or nacl builders. The coordinator still runs those under Docker directly. A future change will convert those to VM images as well. Change-Id: Iedb136ae3daf888c955eb843bdcc9a638d08f5e9 Reviewed-on: https://go-review.googlesource.com/3341 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-28 01:22:21 +03:00
addBuilder(BuildConfig{
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
Name: "linux-amd64-race",
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
HostType: "host-linux-jessie",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
dashboard: don't build x/build on so many builders It was kinda overkill. Change-Id: Ied7fc42aa59a1ed18cc1da20e51c8ed386d84629 Reviewed-on: https://go-review.googlesource.com/c/build/+/210378 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-07 08:14:28 +03:00
buildsRepo: defaultPlusExpBuild,
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
distTestAdjust: fasterTrybots,
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
numTestHelpers: 1,
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
numTryTestHelpers: 5,
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
},
all: add env support, freebsd-386, freebsd-amd64-race, re-enable plan9 Update golang/go#9491 Update golang/go#8639 Change-Id: I1f43c751777f10a6d5870ca9bbb8c2a3430189bf Reviewed-on: https://go-review.googlesource.com/3170 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-22 02:15:48 +03:00
})
env: convert linux-x86-clang to be VM-based Change-Id: I583d751652dec9176169b4d4705f3813506017e3 Reviewed-on: https://go-review.googlesource.com/4790 Reviewed-by: Minux Ma <minux@golang.org>
2015-02-13 06:45:02 +03:00
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "linux-386-clang",
HostType: "host-linux-clang",
Notes: "Debian jessie + clang 3.9 instead of gcc",
env: []string{"CC=/usr/bin/clang", "GOHOSTARCH=386"},
env: convert linux-x86-clang to be VM-based Change-Id: I583d751652dec9176169b4d4705f3813506017e3 Reviewed-on: https://go-review.googlesource.com/4790 Reviewed-by: Minux Ma <minux@golang.org>
2015-02-13 06:45:02 +03:00
})
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "linux-amd64-clang",
HostType: "host-linux-clang",
Notes: "Debian jessie + clang 3.9 instead of gcc",
env: []string{"CC=/usr/bin/clang"},
env: convert linux-x86-clang to be VM-based Change-Id: I583d751652dec9176169b4d4705f3813506017e3 Reviewed-on: https://go-review.googlesource.com/4790 Reviewed-by: Minux Ma <minux@golang.org>
2015-02-13 06:45:02 +03:00
})
env: update linux-x86-sid to be a VM Fixes #9873 Change-Id: I08c501ef52ebde4cdbc48fe45a5342488000c433 Reviewed-on: https://go-review.googlesource.com/4874 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-14 06:01:32 +03:00
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "linux-386-sid",
HostType: "host-linux-sid",
Notes: "Debian sid (unstable)",
env: []string{"GOHOSTARCH=386"},
env: update linux-x86-sid to be a VM Fixes #9873 Change-Id: I08c501ef52ebde4cdbc48fe45a5342488000c433 Reviewed-on: https://go-review.googlesource.com/4874 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-14 06:01:32 +03:00
})
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "linux-amd64-sid",
HostType: "host-linux-sid",
Notes: "Debian sid (unstable)",
env: update linux-x86-sid to be a VM Fixes #9873 Change-Id: I08c501ef52ebde4cdbc48fe45a5342488000c433 Reviewed-on: https://go-review.googlesource.com/4874 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-02-14 06:01:32 +03:00
})
env/linux-x86-fedora: add Fedora builder Not using a Fedora kernel yet, but better than nothing. Updates golang/go#26796 Change-Id: I228442bf6606696407a8d3fbd098df59df7353e1 Reviewed-on: https://go-review.googlesource.com/c/build/+/175999 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-08 19:25:38 +03:00
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "linux-amd64-fedora",
HostType: "host-linux-fedora",
Notes: "Fedora",
env/linux-x86-fedora: add Fedora builder Not using a Fedora kernel yet, but better than nothing. Updates golang/go#26796 Change-Id: I228442bf6606696407a8d3fbd098df59df7353e1 Reviewed-on: https://go-review.googlesource.com/c/build/+/175999 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-08 19:25:38 +03:00
})
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-androidemu",
HostType: "host-android-amd64-emu",
env: []string{
"GOARCH=amd64",
"GOOS=linux",
"CGO_ENABLED=1",
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
tryBot: func(repo, branch, goBranch string) bool {
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
// Only for mobile repo for now, not "go":
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
return repo == "mobile" && branch == "master" && goBranch == "master"
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
return repo == "mobile" && branch == "master" && goBranch == "master"
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
Notes: "Runs GOOS=linux but with the Android emulator attached, for running x/mobile host tests.",
})
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "linux-amd64-jessie",
HostType: "host-linux-jessie",
Notes: "Debian Jessie. The normal 'linux-amd64' builder is stretch. We use Jessie for our release builds due to https://golang.org/issue/31293",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
env: []string{
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
dashboard: enable outbound firewall on a few more Linux-based builders Updates golang/go#30612 Change-Id: Id472e753043976b8671c15491b82bb57e3804d95 Reviewed-on: https://go-review.googlesource.com/c/build/+/165638 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-06 08:20:12 +03:00
},
dashboard, env: add Debian Stretch builder, rename Jessie builder Updates golang/go#25386 Change-Id: I65e3dda63e5ea86fa94480e3305556b73afe121a Reviewed-on: https://go-review.googlesource.com/113059 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-14 20:22:45 +03:00
})
dashboard: add "longtest" builder to run all tests, without -short mode Fixes golang/go#12508 Change-Id: I62e1e296f10c7622c1967a5530c4b0e07c136dfd Reviewed-on: https://go-review.googlesource.com/113436 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-16 22:50:39 +03:00
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "linux-amd64-longtest",
HostType: "host-linux-stretch-morecpu",
Notes: "Debian Stretch with go test -short=false",
dashboard: include longtest builders for trybots on release branches The goal of this test policy change is to decrease the chance of a Go repository release branch from getting into a state where some longtest post-submit builders are failing, as this will interfere with the ability to issue releases after golang/go#29252 is fixed. There are not that many CLs sent to release branches, so the extra latency caused by slower longtest builders being a part of trybots should not be very disruptive. For golang/go#37827. For golang/go#29252. Change-Id: I819b120bb7b763df4acf4583375cdb840793d323 Reviewed-on: https://go-review.googlesource.com/c/build/+/235338 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-05-27 01:19:05 +03:00
tryBot: func(repo, branch, goBranch string) bool {
onReleaseBranch := strings.HasPrefix(branch, "release-branch.")
return repo == "go" && onReleaseBranch // See issue 37827.
},
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
// Test all repos, ignoring buildRepoByDefault.
// For golang.org/x repos, don't test non-latest versions.
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
return repo == "go" || (branch == "master" && goBranch == "master")
},
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
needsGoProxy: true, // for cmd/go module tests
dashboard: increase TIMEOUT_SCALE to 2 for linux-amd64-longtest runtime tests take about 180s in long mode, too close to the timeout limit (180 seconds), and this is causing sporadic timeout failures like https://build.golang.org/log/cda568a39f396e3fee64778a475528be452632ed Bump up the builder's timeout by 2x. Fixes golang/go#25629 Change-Id: I77e0f86f0b236cb953457ebdcbfc5763c2c6d54b Reviewed-on: https://go-review.googlesource.com/115016 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-29 22:03:23 +03:00
env: []string{
dashboard: increase timeout scale for longtest builders, add more CPU Updates golang/go#25886 Change-Id: I5168e291ab77cbd3843bdc39e319a68dfa65aedd Reviewed-on: https://go-review.googlesource.com/c/build/+/167638 Reviewed-by: Russ Cox <rsc@golang.org>
2019-03-14 19:06:57 +03:00
"GO_TEST_TIMEOUT_SCALE=5", // give them lots of time
dashboard: increase TIMEOUT_SCALE to 2 for linux-amd64-longtest runtime tests take about 180s in long mode, too close to the timeout limit (180 seconds), and this is causing sporadic timeout failures like https://build.golang.org/log/cda568a39f396e3fee64778a475528be452632ed Bump up the builder's timeout by 2x. Fixes golang/go#25629 Change-Id: I77e0f86f0b236cb953457ebdcbfc5763c2c6d54b Reviewed-on: https://go-review.googlesource.com/115016 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-29 22:03:23 +03:00
},
dashboard: try to speed up pre-submit longtest builders The longtest builders are currently primarily post-submit builders, where it's okay for them to be as slow as they need to be in order to provide additional test coverage. In this context, whether they take 40 minutes or 50 makes little difference. The longtest builders are also sometimes requested via SlowBots for changes that are riskier than usual, or otherwise desire additional coverage beyond the normal TryBots. They're also always enabled for CLs to release branches. In such contexts, speeding up SlowBot runs from 40 minutes to 20 or less would be appreciated and in turn help people use longtest SlowBots more frequently. Longtest builders are already configured to use sharded tests. Configure them to use additional helpers to speed up test execution. Try out 3, 5, and 9 helpers to see how much it helps before settling. For golang/go#37439. Change-Id: I425bc0257b7a54bb32c0eb1719fea7ba3f4fd461 Reviewed-on: https://go-review.googlesource.com/c/build/+/268037 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-11-06 15:55:36 +03:00
// TODO(golang.org/issue/37439): Decide whether it's worth it
// to go for 3~9 extra helpers for -longtest SlowBot requests
// based on how well having more helpers scales. Try 3, 5, 9.
numTryTestHelpers: 9,
dashboard: add "longtest" builder to run all tests, without -short mode Fixes golang/go#12508 Change-Id: I62e1e296f10c7622c1967a5530c4b0e07c136dfd Reviewed-on: https://go-review.googlesource.com/113436 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-05-16 22:50:39 +03:00
})
dashboard: add windows and 32-bit longtest builders Fixes golang/go#29641 Fixes golang/go#26529 Change-Id: I5ab3aa443276ab36b866d7fabc1d78a82b34cb64 Reviewed-on: https://go-review.googlesource.com/c/build/+/202479 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 19:58:47 +03:00
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "linux-386-longtest",
HostType: "host-linux-stretch-morecpu",
Notes: "Debian Stretch with go test -short=false; to get 32-bit coverage",
dashboard: include longtest builders for trybots on release branches The goal of this test policy change is to decrease the chance of a Go repository release branch from getting into a state where some longtest post-submit builders are failing, as this will interfere with the ability to issue releases after golang/go#29252 is fixed. There are not that many CLs sent to release branches, so the extra latency caused by slower longtest builders being a part of trybots should not be very disruptive. For golang/go#37827. For golang/go#29252. Change-Id: I819b120bb7b763df4acf4583375cdb840793d323 Reviewed-on: https://go-review.googlesource.com/c/build/+/235338 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-05-27 01:19:05 +03:00
tryBot: func(repo, branch, goBranch string) bool {
onReleaseBranch := strings.HasPrefix(branch, "release-branch.")
return repo == "go" && onReleaseBranch // See issue 37827.
},
dashboard: add windows and 32-bit longtest builders Fixes golang/go#29641 Fixes golang/go#26529 Change-Id: I5ab3aa443276ab36b866d7fabc1d78a82b34cb64 Reviewed-on: https://go-review.googlesource.com/c/build/+/202479 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 19:58:47 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
b := buildRepoByDefault(repo)
if repo != "go" && !(branch == "master" && goBranch == "master") {
// For golang.org/x repos, don't test non-latest versions.
b = false
dashboard: stop testing x/build on even more builders Follow up to CL 210378 after looking at: https://build.golang.org/?repo=golang.org%2fx%2fbuild Updates golang/go#30640 Change-Id: I81505a9ca2985ffa6e75922665b14ef4de2af860 Reviewed-on: https://go-review.googlesource.com/c/build/+/211100 Reviewed-by: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-12-12 20:43:59 +03:00
}
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
return b
dashboard: add windows and 32-bit longtest builders Fixes golang/go#29641 Fixes golang/go#26529 Change-Id: I5ab3aa443276ab36b866d7fabc1d78a82b34cb64 Reviewed-on: https://go-review.googlesource.com/c/build/+/202479 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 19:58:47 +03:00
},
needsGoProxy: true, // for cmd/go module tests
env: []string{
dashboard: set GO{,HOST}ARCH in linux-386-longtest builder The linux-386-longtest builder was added in CL 202479 with these environment variables unset, which caused the builder to run long tests for the linux/amd64 pair. Setting these to "386" makes the builder run long tests for linux/386 instead, which increases 32-bit test coverage as originally intended. For golang/go#29641. Change-Id: I9fbebc9e37e5f27f21d840d86006192daeaee8c5 Reviewed-on: https://go-review.googlesource.com/c/build/+/234520 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-19 02:34:15 +03:00
"GOARCH=386",
"GOHOSTARCH=386",
dashboard: add windows and 32-bit longtest builders Fixes golang/go#29641 Fixes golang/go#26529 Change-Id: I5ab3aa443276ab36b866d7fabc1d78a82b34cb64 Reviewed-on: https://go-review.googlesource.com/c/build/+/202479 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 19:58:47 +03:00
"GO_TEST_TIMEOUT_SCALE=5", // give them lots of time
},
dashboard: try to speed up pre-submit longtest builders The longtest builders are currently primarily post-submit builders, where it's okay for them to be as slow as they need to be in order to provide additional test coverage. In this context, whether they take 40 minutes or 50 makes little difference. The longtest builders are also sometimes requested via SlowBots for changes that are riskier than usual, or otherwise desire additional coverage beyond the normal TryBots. They're also always enabled for CLs to release branches. In such contexts, speeding up SlowBot runs from 40 minutes to 20 or less would be appreciated and in turn help people use longtest SlowBots more frequently. Longtest builders are already configured to use sharded tests. Configure them to use additional helpers to speed up test execution. Try out 3, 5, and 9 helpers to see how much it helps before settling. For golang/go#37439. Change-Id: I425bc0257b7a54bb32c0eb1719fea7ba3f4fd461 Reviewed-on: https://go-review.googlesource.com/c/build/+/268037 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-11-06 15:55:36 +03:00
// TODO(golang.org/issue/37439): Decide whether it's worth it
// to go for 3~9 extra helpers for -longtest SlowBot requests
// based on how well having more helpers scales. Try 3, 5, 9.
numTryTestHelpers: 5,
dashboard: add windows and 32-bit longtest builders Fixes golang/go#29641 Fixes golang/go#26529 Change-Id: I5ab3aa443276ab36b866d7fabc1d78a82b34cb64 Reviewed-on: https://go-review.googlesource.com/c/build/+/202479 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 19:58:47 +03:00
})
dashboard: add linux-arm-qemu builder (disabled) and adjust plan9 builder Change-Id: If2b4074f9bc1b44e97897096eb173b42c8323250 Reviewed-on: https://go-review.googlesource.com/6411 Reviewed-by: Salman Aljammaz <salman.aljammaz@gmail.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-03-01 20:23:57 +03:00
addBuilder(BuildConfig{
dashboard, cmd/coordinator: move cross compilation config into dashboard package The dashboard package has been gradually absorbing all configuration, keeping cmd/coordinator only about mechanics. But we'd missed this cross compilation bit in earlier cleanups. Move it, and add some tests & docs while I'm at it. Also, disable the linux-arm-nativemake builder by default. It's redundant with linux-arm. They used to be more different when we always cross compiled for linux-arm, but I guess that got disabled at some point. At this poin it serves no purpose except as an opt-in slowbot if the user really wants to run make.bash on scaleway. Change-Id: Iac683ccf4d30214330480f9db39f131263b1bf8a Reviewed-on: https://go-review.googlesource.com/c/build/+/202020 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-10-19 05:49:53 +03:00
Name: "linux-arm",
HostType: "host-linux-arm-scaleway",
CrossCompileConfig: &CrossCompileConfig{
CompileHostType: "host-linux-armhf-cross",
CCForTarget: "arm-linux-gnueabihf-gcc",
GOARM: "7",
AlwaysCrossCompile: false,
},
cmd/coordinator: warn about known linux-arm SlowBot issue The current linux-arm builder is known to have trouble when used as a SlowBot. Start warning about it when the builder is requested via the TRY= SlowBot UI. I've considered also removing or disabling the "arm" SlowBot alias, but that would make it easier to miss that there's an issue, since SlowBots don't warn about unknown builders: If you specify an unknown TRY= token, it'll just ignore it and won't report an error. We can consider making further changes as this situation evolves. The goal here is to start notifying about a known problem sooner. For golang/go#35628. For golang/go#40872. Change-Id: Ibc1205720c44ec4823c632c04fc2f887368258c1 Reviewed-on: https://go-review.googlesource.com/c/build/+/249420 Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-08-20 02:15:20 +03:00
tryBot: nil, // Issues #22748, #22749, #35628, #40872.
KnownIssue: 35628,
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
FlakyNet: true,
numTestHelpers: 2,
numTryTestHelpers: 7,
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
distTestAdjust: func(run bool, distTest string, isNormalTry bool) bool {
dashboard: skip some slow tests on linux-arm Fixes golang/go#35543 Change-Id: I2554d6ef8400201f10f5f0f0a45a5bbd53104f95 Reviewed-on: https://go-review.googlesource.com/c/build/+/206799 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-12 21:28:02 +03:00
switch distTest {
case "api", "reboot":
return false
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
dashboard: skip some slow tests on linux-arm Fixes golang/go#35543 Change-Id: I2554d6ef8400201f10f5f0f0a45a5bbd53104f95 Reviewed-on: https://go-review.googlesource.com/c/build/+/206799 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-12 21:28:02 +03:00
},
dashboard, cmd/coordinator: sharded ARM trybot-only builders This is a quick & hacky version of our eventual sharding plan. Later we want to run make.bash once, snapshot it, and then intelligently schedule all the tests out over N machines (starting with the built snapshot), giving each the next test to run as they become idle. And then we want to stitch together all of their output. But this CL just makes 6 ARM builders, each of which runs make.bash (4 minutes) and then each which runs a statically-configured subset of the tests. Based on measurements, each should run between 8.2 and 10.6 minutes. We can break up some of the larger test cases later into more shardable pieces (in particular: "test"). Update golang/go#10029 Change-Id: Ib3e008b61ce6f28d799ae298c8467bf62d89e5d4 Reviewed-on: https://go-review.googlesource.com/10056 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-05-14 23:39:58 +03:00
})
dashboard: add linux-arm and linux-arm-arm5 builders Change-Id: Ib8997f76bdf7965d34543cbb6221d6bdb25246fa Reviewed-on: https://go-review.googlesource.com/9840 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2015-05-07 21:36:25 +03:00
addBuilder(BuildConfig{
dashboard: add a make.bash-only builder on real ARM hardware This is a new builder in prep for the change to the "linux-arm" builder where the GOARCH=arm make.bash will be cross-compiled from a Kubernetes container on fast hardware. Updates golang/go#17105 (cross-compile ARM builders' make.bash) Updates golang/go#17104 (5 minute trybots) Change-Id: Icfd2644d77639f731151abe54839322960418254 Reviewed-on: https://go-review.googlesource.com/29670 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 02:29:31 +03:00
Name: "linux-arm-nativemake",
Notes: "runs make.bash on real ARM hardware, but does not run tests",
dashboard: fix build Tests (and binaries) panicked before this. Change-Id: If984e9e30016d403af2a94971fc4a419a269b73a Reviewed-on: https://go-review.googlesource.com/46510 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-06-23 17:02:38 +03:00
HostType: "host-linux-arm-scaleway",
dashboard, cmd/coordinator: move cross compilation config into dashboard package The dashboard package has been gradually absorbing all configuration, keeping cmd/coordinator only about mechanics. But we'd missed this cross compilation bit in earlier cleanups. Move it, and add some tests & docs while I'm at it. Also, disable the linux-arm-nativemake builder by default. It's redundant with linux-arm. They used to be more different when we always cross compiled for linux-arm, but I guess that got disabled at some point. At this poin it serves no purpose except as an opt-in slowbot if the user really wants to run make.bash on scaleway. Change-Id: Iac683ccf4d30214330480f9db39f131263b1bf8a Reviewed-on: https://go-review.googlesource.com/c/build/+/202020 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-10-19 05:49:53 +03:00
tryOnly: true,
tryBot: nil,
dashboard: add a make.bash-only builder on real ARM hardware This is a new builder in prep for the change to the "linux-arm" builder where the GOARCH=arm make.bash will be cross-compiled from a Kubernetes container on fast hardware. Updates golang/go#17105 (cross-compile ARM builders' make.bash) Updates golang/go#17104 (5 minute trybots) Change-Id: Icfd2644d77639f731151abe54839322960418254 Reviewed-on: https://go-review.googlesource.com/29670 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-23 02:29:31 +03:00
StopAfterMake: true,
})
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
addBuilder(BuildConfig{
Name: "linux-arm-arm5spacemonkey",
HostType: "host-linux-arm5spacemonkey",
dashboard, cmd/coordinator: move cross compilation config into dashboard package The dashboard package has been gradually absorbing all configuration, keeping cmd/coordinator only about mechanics. But we'd missed this cross compilation bit in earlier cleanups. Move it, and add some tests & docs while I'm at it. Also, disable the linux-arm-nativemake builder by default. It's redundant with linux-arm. They used to be more different when we always cross compiled for linux-arm, but I guess that got disabled at some point. At this poin it serves no purpose except as an opt-in slowbot if the user really wants to run make.bash on scaleway. Change-Id: Iac683ccf4d30214330480f9db39f131263b1bf8a Reviewed-on: https://go-review.googlesource.com/c/build/+/202020 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-10-19 05:49:53 +03:00
CrossCompileConfig: &CrossCompileConfig{
CompileHostType: "host-linux-armel-cross",
CCForTarget: "arm-linux-gnueabi-gcc",
GOARM: "5",
AlwaysCrossCompile: true,
},
dashboard: increase arm5 builder timeouts Change-Id: I1c1af4f5f6d98ca78da2ab27c3006da867492777 Reviewed-on: https://go-review.googlesource.com/82935 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-12-08 20:39:57 +03:00
env: []string{
"GOARM=5",
"GO_TEST_TIMEOUT_SCALE=4", // arm is normally 2; double that.
},
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
return branch == "master" && goBranch == "master" && buildRepoByDefault(repo)
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
},
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
distTestAdjust: func(run bool, distTest string, isNormalTry bool) bool {
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
if strings.Contains(distTest, "vendor/github.com/google/pprof") {
// Not worth it. And broken.
return false
}
dashboard: skip some slow/broken tests on arm5 builder Updates golang/go#24754 Change-Id: Ie5e94a8fd0b76e6394249a734d8fc1a61650dbad Reviewed-on: https://go-review.googlesource.com/106118 Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2018-04-10 23:03:26 +03:00
if distTest == "api" {
// Broken on this build config (Issue
// 24754), and not worth it on slow
// builder. It's covered by other
// builders anyway.
return false
}
if strings.HasPrefix(distTest, "test:") {
// Slow, and not worth it on slow builder.
return false
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
},
cmd/buildlet, dashboard: add linux-arm-arm5 buildlet Updates golang/go#17082 Updates golang/go#19662 Change-Id: I2b8294f9a8d9144033f8a3479a64186b0f5143a2 Reviewed-on: https://go-review.googlesource.com/38453 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-03-22 23:53:27 +03:00
})
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
addBuilder(BuildConfig{
dashboard: speed up js/wasm trybots We don't need to run all the tests in a trybot. Run just enough to get good coverage, without going over 5 minutes. Any regressions elsewhere will be caught by the build.golang.org (slower) runs. Change-Id: I32f1fc17681bfb509844d2bd35b05c950806d283 Reviewed-on: https://go-review.googlesource.com/121938 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-28 23:30:35 +03:00
Name: "js-wasm",
HostType: "host-js-wasm",
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: explicitTrySet("go"),
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
switch repo {
case "go":
return true
dashboard: stop testing x/build on even more builders Follow up to CL 210378 after looking at: https://build.golang.org/?repo=golang.org%2fx%2fbuild Updates golang/go#30640 Change-Id: I81505a9ca2985ffa6e75922665b14ef4de2af860 Reviewed-on: https://go-review.googlesource.com/c/build/+/211100 Reviewed-by: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-12-12 20:43:59 +03:00
case "build", "mobile", "exp", "benchmarks", "debug", "perf", "talks", "tools", "tour", "website":
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
return false
default:
return branch == "master" && goBranch == "master"
}
dashboard: update some build policy Mostly some regressions (building too much) from the massive refactoring in CL 166218, but also some minor changes. See test updates. Change-Id: Ie30f51b425367f18a49764ac971e262131959bc8 Reviewed-on: https://go-review.googlesource.com/c/build/+/166557 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 07:20:46 +03:00
},
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
distTestAdjust: func(run bool, distTest string, isNormalTry bool) bool {
if isNormalTry {
dashboard: speed up js/wasm trybots We don't need to run all the tests in a trybot. Run just enough to get good coverage, without going over 5 minutes. Any regressions elsewhere will be caught by the build.golang.org (slower) runs. Change-Id: I32f1fc17681bfb509844d2bd35b05c950806d283 Reviewed-on: https://go-review.googlesource.com/121938 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-28 23:30:35 +03:00
if strings.Contains(distTest, "/internal/") ||
strings.Contains(distTest, "vendor/golang.org/x/arch") {
return false
}
switch distTest {
case "cmd/go", "nolibgcc:crypto/x509":
return false
}
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
dashboard: speed up js/wasm trybots We don't need to run all the tests in a trybot. Run just enough to get good coverage, without going over 5 minutes. Any regressions elsewhere will be caught by the build.golang.org (slower) runs. Change-Id: I32f1fc17681bfb509844d2bd35b05c950806d283 Reviewed-on: https://go-review.googlesource.com/121938 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-28 23:30:35 +03:00
},
dashboard: re-enable js/wasm test dir tests They were removed to make the build faster but they've let a few bugs slip by now. Re-enable. The 10 test shards adds 812 seconds of test time. Over 5 machines, that's 162.4 seconds each. Add another test shard to bring it down more. js-wasm trybots are currently at ~350 seconds, so we have some budget to get a bit slower and still be under 5 minutes. Updates golang/go#34768 Change-Id: Ic40c226d26e873227510e644756b0cc969151186 Reviewed-on: https://go-review.googlesource.com/c/build/+/200477 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-10 22:09:40 +03:00
numTryTestHelpers: 5,
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
env: []string{
"GOOS=js", "GOARCH=wasm", "GOHOSTOS=linux", "GOHOSTARCH=amd64",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/workdir/go/misc/wasm",
cmd/coordinator: follow module proxy redirects And because this uses new-in-Go1.13 http.Header.Clone, restrict all the files to Go 1.13+ and bump the Dockerfile Go version. Fixes golang/go#33928 Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132 Reviewed-on: https://go-review.googlesource.com/c/build/+/192341 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-29 21:28:01 +03:00
"GO_DISABLE_OUTBOUND_NETWORK=1",
env/js-wasm, dashboard: add start of a js-wasm builder Updates golang/go#18892 Change-Id: I4b16afe8f0ce74f6ecd022f8bca14ada94ab308f Reviewed-on: https://go-review.googlesource.com/112736 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-11 08:54:32 +03:00
},
})
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "openbsd-386-62",
HostType: "host-openbsd-386-62",
distTestAdjust: noTestDirAndNoReboot,
dashboard: give openbsd/386 more time, and skip pprof vendor tests on arm5 Updates golang/go#19547 Updates golang/go#22594 Change-Id: I0200eebf39c1fada6c6ce146afacb9d140142c5b Reviewed-on: https://go-review.googlesource.com/84035 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-14 21:28:16 +03:00
env: []string{
// cmd/go takes ~192 seconds on openbsd-386
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
env/openbsd, dashboard: add OpenBSD 6.2 386 builder Fixes golang/go#22835 Change-Id: I2763035fd60bf348d8ccc532045c9cf33c405a52 Reviewed-on: https://go-review.googlesource.com/81875 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 08:22:36 +03:00
})
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
addBuilder(BuildConfig{
Name: "openbsd-amd64-62",
HostType: "host-openbsd-amd64-62",
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
distTestAdjust: noTestDirAndNoReboot,
dashboard: switch OpenBSD trybot from 6.2 to 6.4 OpenBSD 6.4 is the current OpenBSD release, while 6.2 has just stopped being supported. The new 6.4 builder has proven to be stable and reliable, so start using it for trybots as well. Looking at the non-trybot build times for the Go repository with OpenBSD 6.2 and 6.4 builders to get a sense of the build speed, it seems OpenBSD 6.4 is about 15% slower (1200~ seconds on 6.2 vs 1380~ seconds on 6.4). This might be due to security-oriented changes in OpenBSD 6.3 and 6.4 that were done to mitigate Spectre/Meltdown vulnerabilities. CL 145022 has helped recover some lost performance by re-enabling SMT. OpenBSD trybots seem to take around 6 minutes now, so there's some room to make them slower before they're the bottleneck. Try to update for now, and if it becomes a bottleneck, then we can prioritize investigating ways to improve OpenBSD 6.4 builder performance further if possible. References: • https://www.openbsd.org/64.html • https://www.openbsd.org/63.html • https://man.openbsd.org/sysctl.2#HW_SMT_2 Fixes golang/go#26833 Change-Id: I42c136fab800df46866a95017a8f70047b37f847 Reviewed-on: https://go-review.googlesource.com/c/144777 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 02:07:13 +03:00
tryBot: nil,
dashboard: add openbsd-amd64-62 (OpenBSD 6.2) builder 386 isn't ready yet. Updates golang/go#22835 Change-Id: Id452140492c0bc5ee850703e640abcbd41e75f9c Reviewed-on: https://go-review.googlesource.com/79835 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-24 21:14:33 +03:00
numTestHelpers: 0,
numTryTestHelpers: 5,
})
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
addBuilder(BuildConfig{
Name: "openbsd-amd64-64",
HostType: "host-openbsd-amd64-64",
dashboard, cmd/coordinator, maintner/maintnerd: add support for BuildConfig.MinimumGoVersion field The new BuildConfig.MinimumGoVersion field specifies the minimum Go version the builder is allowed to use. It's useful when some of the builders are too new, and do not support all of the supported Go releases (e.g., openbsd-amd64-64 and freebsd-amd64-12_0 currently require Go 1.11 and don't work on Go 1.10). It only needs to be set when a builder doesn't support all supported Go releases, since we don't typically test unsupported Go releases. To allow cmd/coordinator to use this field and filter out work it receives from maintner/maintnerd's GoFindTryWork RPC call, we add a GoVersion slice to apipb.GerritTryWorkItem, and populate it in maintapi.apiService.GoFindTryWork method. For trybots on the Go repo, the GoVersion field is determined from the branch name. For "release-branch.goX.Y" branches, it parses out the major-minor Go version from the branch name. For master and other branches, it assumes the latest Go release. For trybots on subrepos, we already have the Go version information available, so use it directly. Afterwards, all that's left is to modify newTrySet in cmd/coordinator to make use of BuildConfig.MinimumGoVersion and work.GoVersion to skip builders that are too new for the Go version that needs to be tested. Fixes golang/go#29265 Change-Id: I50b01830647e33e37e9eb8b89e0f2518812fa44f Reviewed-on: https://go-review.googlesource.com/c/155463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-01-12 00:44:01 +03:00
MinimumGoVersion: types.MajorMinor{1, 11},
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
distTestAdjust: noTestDirAndNoReboot,
dashboard: switch OpenBSD trybot from 6.2 to 6.4 OpenBSD 6.4 is the current OpenBSD release, while 6.2 has just stopped being supported. The new 6.4 builder has proven to be stable and reliable, so start using it for trybots as well. Looking at the non-trybot build times for the Go repository with OpenBSD 6.2 and 6.4 builders to get a sense of the build speed, it seems OpenBSD 6.4 is about 15% slower (1200~ seconds on 6.2 vs 1380~ seconds on 6.4). This might be due to security-oriented changes in OpenBSD 6.3 and 6.4 that were done to mitigate Spectre/Meltdown vulnerabilities. CL 145022 has helped recover some lost performance by re-enabling SMT. OpenBSD trybots seem to take around 6 minutes now, so there's some room to make them slower before they're the bottleneck. Try to update for now, and if it becomes a bottleneck, then we can prioritize investigating ways to improve OpenBSD 6.4 builder performance further if possible. References: • https://www.openbsd.org/64.html • https://www.openbsd.org/63.html • https://man.openbsd.org/sysctl.2#HW_SMT_2 Fixes golang/go#26833 Change-Id: I42c136fab800df46866a95017a8f70047b37f847 Reviewed-on: https://go-review.googlesource.com/c/144777 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 02:07:13 +03:00
tryBot: defaultTrySet(),
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
numTestHelpers: 0,
numTryTestHelpers: 5,
})
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "openbsd-386-64",
HostType: "host-openbsd-386-64",
tryBot: explicitTrySet("sys"),
distTestAdjust: noTestDirAndNoReboot,
dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders This change adds amd64 and 386 OpenBSD 6.4 snapshot builders. The snapshots are from October 4, 2018: /pub/OpenBSD/snapshots/amd64/BUILDINFO: Build date: 1538669162 - Thu Oct 4 16:06:02 UTC 2018 /pub/OpenBSD/snapshots/i386/BUILDINFO: Build date: 1538681766 - Thu Oct 4 19:36:06 UTC 2018 These are meant to be replaced with the final version of OpenBSD 6.4 when it is released. Modify make.bash to support fetching snapshot versions. The amd64 architecture image was tested with cmd/debugnewvm and passed all tests successfully. Updates golang/go#26833. Change-Id: Iea456605ace08f25d4eba37f9a962e6b9af95369 Reviewed-on: https://go-review.googlesource.com/c/139857 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-05 01:14:50 +03:00
})
dashboard: add openbsd-arm-jsing builder This will allow the openbsd-arm builder to be migrated to a buildlet. Change-Id: If38788daf2d1e7f237ebdcce384a098e08bffb0b Reviewed-on: https://go-review.googlesource.com/c/build/+/192598 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-30 20:09:05 +03:00
addBuilder(BuildConfig{
dashboard: reduce the amount of work for openbsd/arm Given the performance of this hardware, reduce the amount of work that it is trying to do. Updates golang/go#25498 Change-Id: I943ab2b01bd10c5217c097464d3904b0f4de2d0a Reviewed-on: https://go-review.googlesource.com/c/build/+/228597 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-04-16 18:47:53 +03:00
Name: "openbsd-arm-jsing",
HostType: "host-openbsd-arm-joelsing",
SkipSnapshot: true,
buildsRepo: func(repo, branch, goBranch string) bool {
switch repo {
dashboard: add openbsd-arm64-jsing builder (reverse) Update golang/go#31656 Fixes golang/go#34035 Change-Id: I7e3d20be148ad5111efeb78702352da14274938e Reviewed-on: https://go-review.googlesource.com/c/build/+/250117 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-20 21:42:01 +03:00
case "go", "net", "sys":
return branch == "master" && goBranch == "master"
default:
return false
}
},
distTestAdjust: noTestDirAndNoReboot,
tryBot: nil,
env: []string{
// The machine is slow.
"GO_TEST_TIMEOUT_SCALE=5",
},
})
addBuilder(BuildConfig{
Name: "openbsd-arm64-jsing",
HostType: "host-openbsd-arm64-joelsing",
SkipSnapshot: true,
buildsRepo: func(repo, branch, goBranch string) bool {
switch repo {
dashboard: add openbsd-mips64-jsing builder (reverse) Update golang/go#40995 Change-Id: Ie733438ab1823d65109e684a6a5e3098cdfdb252 Reviewed-on: https://go-review.googlesource.com/c/build/+/250118 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-23 19:02:40 +03:00
case "go", "net", "sys":
return branch == "master" && goBranch == "master"
default:
return false
}
},
distTestAdjust: noTestDirAndNoReboot,
tryBot: nil,
env: []string{
// The machine is slow.
"GO_TEST_TIMEOUT_SCALE=5",
},
})
addBuilder(BuildConfig{
Name: "openbsd-mips64-jsing",
HostType: "host-openbsd-mips64-joelsing",
SkipSnapshot: true,
KnownIssue: 40995,
buildsRepo: func(repo, branch, goBranch string) bool {
switch repo {
dashboard: reduce the amount of work for openbsd/arm Given the performance of this hardware, reduce the amount of work that it is trying to do. Updates golang/go#25498 Change-Id: I943ab2b01bd10c5217c097464d3904b0f4de2d0a Reviewed-on: https://go-review.googlesource.com/c/build/+/228597 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-04-16 18:47:53 +03:00
case "go", "net", "sys":
return branch == "master" && goBranch == "master"
default:
return false
}
},
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
distTestAdjust: noTestDirAndNoReboot,
tryBot: nil,
dashboard: add openbsd-arm-jsing builder This will allow the openbsd-arm builder to be migrated to a buildlet. Change-Id: If38788daf2d1e7f237ebdcce384a098e08bffb0b Reviewed-on: https://go-review.googlesource.com/c/build/+/192598 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-08-30 20:09:05 +03:00
env: []string{
// The machine is slow.
"GO_TEST_TIMEOUT_SCALE=5",
},
})
dashboard, env/netbsd-{amd64,386}: add NetBSD 9.0 builders Update NetBSD amd64 and 386 builders to the latest NetBSD 9.0 release, using the latest versions of anita (2.3) and pkgsrc packages (2019Q4). Also increase the image disk size from 4 GB to 16 GB. The motivation is to help tests that require more disk space and large dependencies. Additional adjustments to the image building script will need to be made to make use of increased disk space for the /tmp volume, as it is currently being mounted in RAM with size equal to 25% of memory. Remove the manual invocation of the mozilla-rootcerts install command. It no longer works as is in NetBSD 9.0. The mozilla-rootcerts-openssl package is added instead, which successfully performs that same task. Hopefully this will be a more robust approach for the future. Continue to use an explicit URL for the NetBSD release download, and remove unused find_latest_release. The plan is to roll out these builders first (without trybots), gain more confidence, enable trybots, and then remove NetBSD 8.0 builders. For golang/go#37244. For golang/go#30839. For golang/go#31858. For golang/go#31726. For golang/go#38283. Change-Id: Id41278eaebe64cfbcf7b1ab9bfd64e550ee0bbf1 Reviewed-on: https://go-review.googlesource.com/c/build/+/226977 Reviewed-by: Benny Siegert <bsiegert@gmail.com> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-02 15:55:30 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "netbsd-amd64-9_0",
HostType: "host-netbsd-amd64-9_0",
distTestAdjust: noTestDirAndNoReboot,
tryBot: explicitTrySet("sys"),
dashboard, env/netbsd-{amd64,386}: add NetBSD 9.0 builders Update NetBSD amd64 and 386 builders to the latest NetBSD 9.0 release, using the latest versions of anita (2.3) and pkgsrc packages (2019Q4). Also increase the image disk size from 4 GB to 16 GB. The motivation is to help tests that require more disk space and large dependencies. Additional adjustments to the image building script will need to be made to make use of increased disk space for the /tmp volume, as it is currently being mounted in RAM with size equal to 25% of memory. Remove the manual invocation of the mozilla-rootcerts install command. It no longer works as is in NetBSD 9.0. The mozilla-rootcerts-openssl package is added instead, which successfully performs that same task. Hopefully this will be a more robust approach for the future. Continue to use an explicit URL for the NetBSD release download, and remove unused find_latest_release. The plan is to roll out these builders first (without trybots), gain more confidence, enable trybots, and then remove NetBSD 8.0 builders. For golang/go#37244. For golang/go#30839. For golang/go#31858. For golang/go#31726. For golang/go#38283. Change-Id: Id41278eaebe64cfbcf7b1ab9bfd64e550ee0bbf1 Reviewed-on: https://go-review.googlesource.com/c/build/+/226977 Reviewed-by: Benny Siegert <bsiegert@gmail.com> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-02 15:55:30 +03:00
})
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "netbsd-386-9_0",
HostType: "host-netbsd-386-9_0",
distTestAdjust: noTestDirAndNoReboot,
dashboard, env/netbsd-{amd64,386}: add NetBSD 9.0 builders Update NetBSD amd64 and 386 builders to the latest NetBSD 9.0 release, using the latest versions of anita (2.3) and pkgsrc packages (2019Q4). Also increase the image disk size from 4 GB to 16 GB. The motivation is to help tests that require more disk space and large dependencies. Additional adjustments to the image building script will need to be made to make use of increased disk space for the /tmp volume, as it is currently being mounted in RAM with size equal to 25% of memory. Remove the manual invocation of the mozilla-rootcerts install command. It no longer works as is in NetBSD 9.0. The mozilla-rootcerts-openssl package is added instead, which successfully performs that same task. Hopefully this will be a more robust approach for the future. Continue to use an explicit URL for the NetBSD release download, and remove unused find_latest_release. The plan is to roll out these builders first (without trybots), gain more confidence, enable trybots, and then remove NetBSD 8.0 builders. For golang/go#37244. For golang/go#30839. For golang/go#31858. For golang/go#31726. For golang/go#38283. Change-Id: Id41278eaebe64cfbcf7b1ab9bfd64e550ee0bbf1 Reviewed-on: https://go-review.googlesource.com/c/build/+/226977 Reviewed-by: Benny Siegert <bsiegert@gmail.com> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-02 15:55:30 +03:00
})
dashboard: add netbsd-arm-bsiegert This seems to be the last builder still using the old builder binary and not buildlet. It needs an entry in here to switch to buildlet and stage0. Change-Id: If2fdd2298ab88d56b6850061050caf27ca7c8983 Reviewed-on: https://go-review.googlesource.com/c/build/+/168061 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-19 16:28:04 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "netbsd-arm-bsiegert",
HostType: "host-netbsd-arm-bsiegert",
distTestAdjust: noTestDirAndNoReboot,
tryBot: nil,
dashboard: add netbsd-arm-bsiegert This seems to be the last builder still using the old builder binary and not buildlet. It needs an entry in here to switch to buildlet and stage0. Change-Id: If2fdd2298ab88d56b6850061050caf27ca7c8983 Reviewed-on: https://go-review.googlesource.com/c/build/+/168061 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-19 16:28:04 +03:00
env: []string{
// The machine is slow.
"GO_TEST_TIMEOUT_SCALE=10",
},
})
dashboard: add netbsd-arm64-bsiegert builder (reverse) Update golang/go#34036 Change-Id: Ic92093e764fa163d97a2f125a4534581adf217da Reviewed-on: https://go-review.googlesource.com/c/build/+/249137 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-08-18 21:42:30 +03:00
addBuilder(BuildConfig{
dashboard: only build tip on netbsd-arm64-bsiegert builder The build fails on 1.15 and earlier. Updates golang/go#30824 Change-Id: I1f62d37cc05dde988cbb7e5124bd97916b38bcb3 Reviewed-on: https://go-review.googlesource.com/c/build/+/257938 Trust: Tobias Klauser <tobias.klauser@gmail.com> Trust: Benny Siegert <bsiegert@gmail.com> Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Benny Siegert <bsiegert@gmail.com>
2020-09-28 11:28:32 +03:00
Name: "netbsd-arm64-bsiegert",
HostType: "host-netbsd-arm64-bsiegert",
buildsRepo: func(repo, branch, goBranch string) bool {
return atLeastGo1(goBranch, 16) && buildRepoByDefault(repo)
},
dashboard: add netbsd-arm64-bsiegert builder (reverse) Update golang/go#34036 Change-Id: Ic92093e764fa163d97a2f125a4534581adf217da Reviewed-on: https://go-review.googlesource.com/c/build/+/249137 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-08-18 21:42:30 +03:00
distTestAdjust: noTestDirAndNoReboot,
tryBot: nil,
env: []string{
// The machine is slow.
"GO_TEST_TIMEOUT_SCALE=10",
},
})
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "plan9-386",
HostType: "host-plan9-386-gce",
dashboard: only test plan9 at master The plan9 port doesn't need to be tested against older Go releases. Its userbase runs at master. Change-Id: I710d98560b234c7555c7be526aa2efdb6473c439 Reviewed-on: https://go-review.googlesource.com/c/build/+/170519 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-03 08:03:37 +03:00
numTestHelpers: 1,
dashboard: disable plan9-386 builder It hasn't passed in months and now spins, wasting resources. Updates golang/go#31261 Updates golang/go#29801 Change-Id: Idcf13ae915bad4febb156c5c5d49f07f76cf9d49 Reviewed-on: https://go-review.googlesource.com/c/build/+/172797 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-18 22:33:19 +03:00
tryOnly: true, // disable it for now; Issue 31261, Issue 29801
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
distTestAdjust: func(run bool, distTest string, isNormalTry bool) bool {
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
switch distTest {
env/plan9: revert SMP changes, stop running cmd/go tests Revert "env: enable SMP in Plan 9 image" This reverts commit 6478d3e7bdff4c8d810ba44f970ca4b0bfc92405. Revert "dashboard: bump plan9 GCE image" This reverts commit ce208996bf954a87d68789bb6a9000256f016c77. Updates golang/go#29822 Change-Id: I47e75d135e092ee9d5a80c66e91c1048c67cd5f6 Reviewed-on: https://go-review.googlesource.com/c/158542 Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: David du Colombier <0intro@gmail.com>
2019-01-19 01:19:58 +03:00
case "api",
"go_test:cmd/go": // takes over 20 minutes without working SMP
return false
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
env/plan9: revert SMP changes, stop running cmd/go tests Revert "env: enable SMP in Plan 9 image" This reverts commit 6478d3e7bdff4c8d810ba44f970ca4b0bfc92405. Revert "dashboard: bump plan9 GCE image" This reverts commit ce208996bf954a87d68789bb6a9000256f016c77. Updates golang/go#29822 Change-Id: I47e75d135e092ee9d5a80c66e91c1048c67cd5f6 Reviewed-on: https://go-review.googlesource.com/c/158542 Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: David du Colombier <0intro@gmail.com>
2019-01-19 01:19:58 +03:00
},
dashboard: don't build x/exp and x/mobile on the plan9 builders Updates golang/go#11811 Change-Id: I776652a3b980772f34d65d8c245a143814aadb6a Reviewed-on: https://go-review.googlesource.com/c/build/+/209338 Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-28 13:33:00 +03:00
buildsRepo: onlyMasterDefault,
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
})
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "windows-amd64-2008",
HostType: "host-windows-amd64-2008",
distTestAdjust: noTestDirAndNoReboot,
buildsRepo: onlyGo,
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
})
cmd/buildlet, dashboard: support for v2 windows images, 386 builder - images now include 32bit gcc in c:\godeps\gcc32 - 64bit gcc has moved to c:\godeps\gcc64 - update host definitions for new image versions - add windows-386-2008 host Updates golang/go#17513 Change-Id: Ib3f5d4b2b1702c05089cf633529ebf38db270a43 Reviewed-on: https://go-review.googlesource.com/42014 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-27 23:12:08 +03:00
addBuilder(BuildConfig{
Name: "windows-386-2008",
HostType: "host-windows-amd64-2008",
dashboard: don't build x/build on so many builders It was kinda overkill. Change-Id: Ied7fc42aa59a1ed18cc1da20e51c8ed386d84629 Reviewed-on: https://go-review.googlesource.com/c/build/+/210378 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-07 08:14:28 +03:00
buildsRepo: defaultPlusExpBuild,
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
distTestAdjust: fasterTrybots,
cmd/buildlet, dashboard: support for v2 windows images, 386 builder - images now include 32bit gcc in c:\godeps\gcc32 - 64bit gcc has moved to c:\godeps\gcc64 - update host definitions for new image versions - add windows-386-2008 host Updates golang/go#17513 Change-Id: Ib3f5d4b2b1702c05089cf633529ebf38db270a43 Reviewed-on: https://go-review.googlesource.com/42014 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-27 23:12:08 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
numTryTestHelpers: 4,
cmd/buildlet, dashboard: support for v2 windows images, 386 builder - images now include 32bit gcc in c:\godeps\gcc32 - 64bit gcc has moved to c:\godeps\gcc64 - update host definitions for new image versions - add windows-386-2008 host Updates golang/go#17513 Change-Id: Ib3f5d4b2b1702c05089cf633529ebf38db270a43 Reviewed-on: https://go-review.googlesource.com/42014 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-27 23:12:08 +03:00
})
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "windows-amd64-2012",
HostType: "host-windows-amd64-2012",
distTestAdjust: noTestDirAndNoReboot,
buildsRepo: onlyGo,
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
dashboard: add configs for new windows 2008, 2012, 2016 builders See also: https://golang.org/cl/41142 Updates golang/go#17513 Change-Id: Ie743ae4604e65892e28423ed0af008450b647197 Reviewed-on: https://go-review.googlesource.com/41393 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2017-04-21 22:35:28 +03:00
})
build: use buildlet on Windows Also adds "gomote rmall" command. Fixes golang/go#8640 Fixes golang/go#9799 Change-Id: I41b62b16df5e855e1367a9cc1c370bb1f928c242 Reviewed-on: https://go-review.googlesource.com/4132 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2015-02-07 04:32:15 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "windows-amd64-2016",
HostType: "host-windows-amd64-2016",
buildsRepo: defaultPlusExpBuild,
distTestAdjust: fasterTrybots,
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2",
},
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
tryBot: defaultTrySet(),
dashboard: update Windows builders, remove old manual Windows builder And clean up some other builders. Change-Id: I1f71836da7008e58d3e76e2cc3170e96cd57ddf6 Reviewed-on: https://go-review.googlesource.com/51432 Reviewed-by: Jeff Johnson <jrjohnson@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-27 07:52:12 +03:00
numTryTestHelpers: 5,
build: use buildlet on Windows Also adds "gomote rmall" command. Fixes golang/go#8640 Fixes golang/go#9799 Change-Id: I41b62b16df5e855e1367a9cc1c370bb1f928c242 Reviewed-on: https://go-review.googlesource.com/4132 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2015-02-07 04:32:15 +03:00
})
dashboard: add windows and 32-bit longtest builders Fixes golang/go#29641 Fixes golang/go#26529 Change-Id: I5ab3aa443276ab36b866d7fabc1d78a82b34cb64 Reviewed-on: https://go-review.googlesource.com/c/build/+/202479 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 19:58:47 +03:00
addBuilder(BuildConfig{
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Name: "windows-amd64-longtest",
dashboard: upsize machine type for windows-amd64-{longtest,race} builders Start using n1-highcpu-8 machine type instead of n1-highcpu-4 for windows-amd64-longtest and windows-amd64-race builders. Continue to use the existing n1-highcpu-4 machine type for all other windows/386 and windows/amd64 builders for now. The primary motivation of this change is to increase amount of RAM available on the longtest builder from 3.6 GB to 7.2 GB, because it is consistenly failing due to being out of memory. Also fix minor typos, and increase consistency in comments and notes. Updates golang/go#33951 Change-Id: Iac5da67fa584059842850a9c8deda98c242741f2 Reviewed-on: https://go-review.googlesource.com/c/build/+/208500 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-11-23 02:01:25 +03:00
HostType: "host-windows-amd64-2016-big",
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
Notes: "Windows Server 2016 with go test -short=false",
dashboard: include longtest builders for trybots on release branches The goal of this test policy change is to decrease the chance of a Go repository release branch from getting into a state where some longtest post-submit builders are failing, as this will interfere with the ability to issue releases after golang/go#29252 is fixed. There are not that many CLs sent to release branches, so the extra latency caused by slower longtest builders being a part of trybots should not be very disruptive. For golang/go#37827. For golang/go#29252. Change-Id: I819b120bb7b763df4acf4583375cdb840793d323 Reviewed-on: https://go-review.googlesource.com/c/build/+/235338 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-05-27 01:19:05 +03:00
tryBot: func(repo, branch, goBranch string) bool {
onReleaseBranch := strings.HasPrefix(branch, "release-branch.")
return repo == "go" && onReleaseBranch // See issue 37827.
},
dashboard: add windows and 32-bit longtest builders Fixes golang/go#29641 Fixes golang/go#26529 Change-Id: I5ab3aa443276ab36b866d7fabc1d78a82b34cb64 Reviewed-on: https://go-review.googlesource.com/c/build/+/202479 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 19:58:47 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
b := defaultPlusExpBuild(repo, branch, goBranch)
if repo != "go" && !(branch == "master" && goBranch == "master") {
// For golang.org/x repos, don't test non-latest versions.
b = false
dashboard: adjust the windows-amd64-longtest policy It shouldn't build more repos than windows-amd64-2016. Fixes golang/go#35185 Change-Id: I310e2ccc521d88692fcf904fe53b4bae94f3f0f0 Reviewed-on: https://go-review.googlesource.com/c/build/+/203885 Reviewed-by: Bryan C. Mills <bcmills@google.com> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-10-28 23:16:36 +03:00
}
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
return b
dashboard: add windows and 32-bit longtest builders Fixes golang/go#29641 Fixes golang/go#26529 Change-Id: I5ab3aa443276ab36b866d7fabc1d78a82b34cb64 Reviewed-on: https://go-review.googlesource.com/c/build/+/202479 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 19:58:47 +03:00
},
needsGoProxy: true, // for cmd/go module tests
env: []string{
"GO_TEST_TIMEOUT_SCALE=5", // give them lots of time
},
dashboard: try to speed up pre-submit longtest builders The longtest builders are currently primarily post-submit builders, where it's okay for them to be as slow as they need to be in order to provide additional test coverage. In this context, whether they take 40 minutes or 50 makes little difference. The longtest builders are also sometimes requested via SlowBots for changes that are riskier than usual, or otherwise desire additional coverage beyond the normal TryBots. They're also always enabled for CLs to release branches. In such contexts, speeding up SlowBot runs from 40 minutes to 20 or less would be appreciated and in turn help people use longtest SlowBots more frequently. Longtest builders are already configured to use sharded tests. Configure them to use additional helpers to speed up test execution. Try out 3, 5, and 9 helpers to see how much it helps before settling. For golang/go#37439. Change-Id: I425bc0257b7a54bb32c0eb1719fea7ba3f4fd461 Reviewed-on: https://go-review.googlesource.com/c/build/+/268037 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-11-06 15:55:36 +03:00
// TODO(golang.org/issue/37439): Decide whether it's worth it
// to go for 3~9 extra helpers for -longtest SlowBot requests
// based on how well having more helpers scales. Try 3, 5, 9.
numTryTestHelpers: 3,
dashboard: add windows and 32-bit longtest builders Fixes golang/go#29641 Fixes golang/go#26529 Change-Id: I5ab3aa443276ab36b866d7fabc1d78a82b34cb64 Reviewed-on: https://go-review.googlesource.com/c/build/+/202479 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 19:58:47 +03:00
})
build: use buildlet on Windows Also adds "gomote rmall" command. Fixes golang/go#8640 Fixes golang/go#9799 Change-Id: I41b62b16df5e855e1367a9cc1c370bb1f928c242 Reviewed-on: https://go-review.googlesource.com/4132 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2015-02-07 04:32:15 +03:00
addBuilder(BuildConfig{
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
Name: "windows-amd64-race",
dashboard: update windows-amd64-race builder to Windows Server 2016 image The purpose of the -race builder is to check for bugs on the Go side, so use the newer available Windows version with the expectation that there should be fewer unfixed bugs left in it. Take this easy step to eliminate a variable for issue golang/go#39196. Also delete host-windows-amd64-2008-big, since it's no longer used. For golang/go#39196. Change-Id: I9095a73ab21a151e5972219bccd5a16b249765b4 Reviewed-on: https://go-review.googlesource.com/c/build/+/235419 Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-27 18:58:11 +03:00
HostType: "host-windows-amd64-2016-big",
dashboard, cmd/coordinator: various fixes * add support for longtest builders for subrepos. * fix race builders for subrepos (they weren't passing the -race flag). * adjust the policy for the js-wasm builders to build fewer subrepos where it'll never work or isn't worth it. * fix the android emu builders which disappeared because an empty string was being passed to buildsRepoAtAll. In some places in the code an empty string for goBranch for the "go" repo meant the same as branch, but I forgot that in the new code, so an old caller was confusing the new config hooks. Rather than make all policy funcs be aware of both ways, the new code in this CL now maps an empty string to the same as the repo's branch when the repo == "go". Adds a test too. * fix some outdated comments. Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7 Reviewed-on: https://go-review.googlesource.com/c/build/+/166897 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-11 21:49:56 +03:00
Notes: "Only runs -race tests (./race.bat)",
dashboard: increase TIMEOUT_SCALE to 2 for windows-amd64 cmd/go takes ~188 seconds on windows-amd64, which is over the default 180 second limit. Bump up the timout rate. Change-Id: Ib0e5289ab6c4d6127d18375e3a72453818b984a8 Reviewed-on: https://go-review.googlesource.com/110415 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-01 00:16:33 +03:00
env: []string{
"GOARCH=amd64",
"GOHOSTARCH=amd64",
// cmd/go takes ~188 seconds on windows-amd64
// now, which is over the 180 second default
// dist test timeout. So, bump this builder
// up:
"GO_TEST_TIMEOUT_SCALE=2"},
build: use buildlet on Windows Also adds "gomote rmall" command. Fixes golang/go#8640 Fixes golang/go#9799 Change-Id: I41b62b16df5e855e1367a9cc1c370bb1f928c242 Reviewed-on: https://go-review.googlesource.com/4132 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2015-02-07 04:32:15 +03:00
})
cmd/coordinator, dashboard: switch to new VMWare-based OS X builders Ignore the old darwin-{amd64,386}-10_10 builders. Don't give them an error, but pretend they don't exist. Also: switch trybots from OS X 10.10 to OS X 10.11, and re-enable sharding. Let's hope for the best. See golang/go#12979. This also enables subrepo tests for all OS X versions. darwin-386-* is currently offline, pending some golang/go#17009 Updates golang/go#9495 (OS X virtualization) Change-Id: I4d53a79087404b5e8051d1aff0c668a92625f442 Reviewed-on: https://go-review.googlesource.com/28583 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-07 08:41:02 +03:00
addBuilder(BuildConfig{
dashboard: disable darwin-amd64-10_11 builder on Go 1.15+ Go 1.15 will require macOS 10.12 and newer¹, so keep the OS X 10.11 builder enabled only on release branches for 1.14 and older. ¹ https://golang.org/doc/go1.14#darwin Fixes golang/go#37425 Change-Id: I89ab0eb56729fe16fb3a524c8f7b9bb25751d708 Reviewed-on: https://go-review.googlesource.com/c/build/+/222797 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-03-10 20:26:20 +03:00
Name: "darwin-amd64-10_11",
HostType: "host-darwin-10_11",
tryBot: nil, // disabled until Macs fixed; https://golang.org/issue/23859
buildsRepo: func(repo, branch, goBranch string) bool {
// Go 1.14 is the last release that will run on macOS 10.11 El Capitan.
// (See https://golang.org/doc/go1.14#darwin.)
return repo == "go" && atMostGo1(branch, 14)
},
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
distTestAdjust: macTestPolicy,
dashboard: fix typo from CL 94601 Sigh. Change-Id: Ife6006f9b0c65fa6ee7be50b6c9f55319e0fa28d Reviewed-on: https://go-review.googlesource.com/94602 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-02-16 02:43:31 +03:00
numTryTestHelpers: 3,
cmd/coordinator, dashboard: switch to new VMWare-based OS X builders Ignore the old darwin-{amd64,386}-10_10 builders. Don't give them an error, but pretend they don't exist. Also: switch trybots from OS X 10.10 to OS X 10.11, and re-enable sharding. Let's hope for the best. See golang/go#12979. This also enables subrepo tests for all OS X versions. darwin-386-* is currently offline, pending some golang/go#17009 Updates golang/go#9495 (OS X virtualization) Change-Id: I4d53a79087404b5e8051d1aff0c668a92625f442 Reviewed-on: https://go-review.googlesource.com/28583 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-09-07 08:41:02 +03:00
})
dashboard: restore darwin-386 buidler Fixes golang/go#17009 Change-Id: Ia9008bcb7b47b7671c158c1a00afc4e765805e4e Reviewed-on: https://go-review.googlesource.com/41617 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-25 02:03:42 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "darwin-386-10_14",
HostType: "host-darwin-10_14",
distTestAdjust: macTestPolicy,
dashboard: require at least Go 1.13 for darwin-386 builder release-branch.go1.12 has failures for darwin-386-10_14, which are fixed in Go 1.13. darwin-386 is not a first-class port and on a deprecated architecture for macOS, and can be skipped on older branches. Fixes golang/go#33756 Change-Id: I5f2e95fe9515978a85b986787d8126655f849629 Reviewed-on: https://go-review.googlesource.com/c/build/+/191878 Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-08-26 23:40:35 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: disable darwin/386 builder on master The 32-bit pairs darwin/386 and darwin/arm are removed in Go 1.15, and the master branch is where Go 1.15 is being developed. The builder has started to fail after CL 226985 (as intended), so remove it. Keep it running on Go 1.14 and 1.13 release branches. Also update the misc-compile-darwin builder description. For golang/go#37610. For golang/go#37611. Change-Id: Ia70386c5ae2821b4397048dcb0862a066463d6de Reviewed-on: https://go-review.googlesource.com/c/build/+/227181 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Andrew Bonventre <andybons@golang.org>
2020-04-04 06:21:35 +03:00
// Go 1.14 is the last release that will support 32-bit binaries on macOS (darwin/386).
// (See https://golang.org/doc/go1.14#darwin.)
return repo == "go" && atMostGo1(branch, 14)
dashboard: require at least Go 1.13 for darwin-386 builder release-branch.go1.12 has failures for darwin-386-10_14, which are fixed in Go 1.13. darwin-386 is not a first-class port and on a deprecated architecture for macOS, and can be skipped on older branches. Fixes golang/go#33756 Change-Id: I5f2e95fe9515978a85b986787d8126655f849629 Reviewed-on: https://go-review.googlesource.com/c/build/+/191878 Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-08-26 23:40:35 +03:00
},
cmd/coordinator: delete useScheduler knob & old pre-scheduler hacks, code, TODOs Updates golang/go#19178 Change-Id: Id4d8b016c41f57bfaba5ee4ab046285607047493 Reviewed-on: https://go-review.googlesource.com/c/build/+/207178 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-14 18:43:33 +03:00
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
})
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "darwin-amd64-10_12",
HostType: "host-darwin-10_12",
distTestAdjust: macTestPolicy,
dashboard: restore darwin-386 buidler Fixes golang/go#17009 Change-Id: Ia9008bcb7b47b7671c158c1a00afc4e765805e4e Reviewed-on: https://go-review.googlesource.com/41617 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-25 02:03:42 +03:00
})
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "darwin-amd64-10_14",
HostType: "host-darwin-10_14",
distTestAdjust: macTestPolicy,
buildsRepo: defaultPlusExp,
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
})
dashboard, cmd/makemac: add macOS 10.15 Catalina builder Added the macOS 10.15 Catalina builder to the dashboard. Catalina builders should use the Mojave guestid in vSphere 6.7 update 3 because the Catalina guestid is not available in this version of vSphere. A Catalina builder will run on this version of vSphere with the Mojave guestid. Modified the expected number of vm instances running on Macstadium due to the addition of macOS 10.15 builders. Updates golang/go#34748 Change-Id: If14f6e088afc05dd1783460f21907e9345e3c338 Reviewed-on: https://go-review.googlesource.com/c/build/+/207937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-11-19 17:55:01 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "darwin-amd64-10_15",
HostType: "host-darwin-10_15",
distTestAdjust: macTestPolicy,
buildsRepo: defaultPlusExpBuild,
dashboard, cmd/makemac: add macOS 10.15 Catalina builder Added the macOS 10.15 Catalina builder to the dashboard. Catalina builders should use the Mojave guestid in vSphere 6.7 update 3 because the Catalina guestid is not available in this version of vSphere. A Catalina builder will run on this version of vSphere with the Mojave guestid. Modified the expected number of vm instances running on Macstadium due to the addition of macOS 10.15 builders. Updates golang/go#34748 Change-Id: If14f6e088afc05dd1783460f21907e9345e3c338 Reviewed-on: https://go-review.googlesource.com/c/build/+/207937 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-11-19 17:55:01 +03:00
})
dashboard: add darwin-amd64-nocgo config, remove nacl-386 trybot Also remove dead nacl-arm. It hasn't run in ages. And update netbsd comment about why 386 doesn't run. And correct its VM image name. Updates golang/go#31705 Updates golang/go#31726 Change-Id: I9de4605f34a052d0a84684fca098388d75602a82 Reviewed-on: https://go-review.googlesource.com/c/build/+/174637 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 01:37:31 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "darwin-amd64-nocgo",
HostType: "host-darwin-10_15",
distTestAdjust: noTestDirAndNoReboot,
env: []string{"CGO_ENABLED=0"},
dashboard: add darwin-amd64-nocgo config, remove nacl-386 trybot Also remove dead nacl-arm. It hasn't run in ages. And update netbsd comment about why 386 doesn't run. And correct its VM image name. Updates golang/go#31705 Updates golang/go#31726 Change-Id: I9de4605f34a052d0a84684fca098388d75602a82 Reviewed-on: https://go-review.googlesource.com/c/build/+/174637 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 01:37:31 +03:00
})
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "darwin-amd64-race",
HostType: "host-darwin-10_15",
distTestAdjust: macTestPolicy,
buildsRepo: onlyGo,
build: add macoS Sierra builder Change-Id: I10a7a9e44a158fdce5287a9b0b065b35d11fb502 Reviewed-on: https://go-review.googlesource.com/35470 Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-19 01:45:12 +03:00
})
dashboard: add Zenly's virtual iOS builders Fixes golang/go#31722 Change-Id: Ia7f0d3f2e1a25000c42a10ff58d7e91d384b1219 Reviewed-on: https://go-review.googlesource.com/c/build/+/174797 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 21:33:26 +03:00
addBuilder(BuildConfig{
dashboard,env: move iOS builders to GOOS=ios Updates golang/go#41610 Change-Id: Ie8cb3294da5566e09fc06c01e679b8b773a439cf Reviewed-on: https://go-review.googlesource.com/c/build/+/259358 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Go Bot <gobot@golang.org> Trust: Elias Naur <mail@eliasnaur.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-10-03 12:37:07 +03:00
Name: "ios-arm64-corellium",
HostType: "host-ios-arm64-corellium-ios",
dashboard: add Zenly's virtual iOS builders Fixes golang/go#31722 Change-Id: Ia7f0d3f2e1a25000c42a10ff58d7e91d384b1219 Reviewed-on: https://go-review.googlesource.com/c/build/+/174797 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 21:33:26 +03:00
Notes: "Virtual iPhone SE running on Corellium; owned by zenly",
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: add Zenly's virtual Android builders Change-Id: Id0be99f343a819c19b0004dc4160175c9bb0ace9 Reviewed-on: https://go-review.googlesource.com/c/build/+/174798 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 23:28:43 +03:00
return repo == "go" && branch == "master" && goBranch == "master"
},
})
addBuilder(BuildConfig{
Name: "android-arm64-corellium",
HostType: "host-android-arm64-corellium-android",
Notes: "Virtual Android running on Corellium; owned by zenly",
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: add Zenly's virtual iOS builders Fixes golang/go#31722 Change-Id: Ia7f0d3f2e1a25000c42a10ff58d7e91d384b1219 Reviewed-on: https://go-review.googlesource.com/c/build/+/174797 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-01 21:33:26 +03:00
return repo == "go" && branch == "master" && goBranch == "master"
},
})
dashboard: add android/arm builder on Corellium Updates golang/go#31722 Change-Id: I07d9f85d47b6cda5eadcfc4f93adff42316f0ff5 Reviewed-on: https://go-review.googlesource.com/c/build/+/174857 TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-02 01:54:54 +03:00
addBuilder(BuildConfig{
Name: "android-arm-corellium",
HostType: "host-android-arm64-corellium-android",
Notes: "Virtual Android running on Corellium; owned by zenly",
buildsRepo: func(repo, branch, goBranch string) bool {
return repo == "go" && branch == "master" && goBranch == "master"
},
env: []string{
"CGO_ENABLED=1",
"GOARCH=arm",
},
})
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
addBuilder(BuildConfig{
Name: "android-386-emu",
HostType: "host-android-amd64-emu", // same amd64 host is used for 386 builder
Notes: "Android emulator on GCE",
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
switch repo {
dashboard: stop testing x/build on even more builders Follow up to CL 210378 after looking at: https://build.golang.org/?repo=golang.org%2fx%2fbuild Updates golang/go#30640 Change-Id: I81505a9ca2985ffa6e75922665b14ef4de2af860 Reviewed-on: https://go-review.googlesource.com/c/build/+/211100 Reviewed-by: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-12-12 20:43:59 +03:00
case "build", "blog", "talks", "review", "tour", "website":
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
return false
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
return atLeastGo1(branch, 13) && atLeastGo1(goBranch, 13)
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
env: []string{
"GOARCH=386",
"GOOS=android",
"GOHOSTARCH=amd64",
"GOHOSTOS=linux",
"CGO_ENABLED=1",
},
})
addBuilder(BuildConfig{
dashboard: add Android trybots Updates golang/go#23824 Change-Id: I6386bc267c35856858b3e46d0998fb60e3406e02 Reviewed-on: https://go-review.googlesource.com/c/build/+/172798 Reviewed-by: Elias Naur <mail@eliasnaur.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-18 22:45:32 +03:00
Name: "android-amd64-emu",
HostType: "host-android-amd64-emu",
Notes: "Android emulator on GCE",
numTryTestHelpers: 3,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
tryBot: func(repo, branch, goBranch string) bool {
dashboard: add android subrepo trybots too Missing from CL 172798 Updates golang/go#23824 Change-Id: I8f1b09ce8f52b555162b344ceddfbfe88e19ad4e Reviewed-on: https://go-review.googlesource.com/c/build/+/174079 Reviewed-by: Elias Naur <mail@eliasnaur.com> Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-26 17:04:00 +03:00
switch repo {
case "go", "mobile", "sys", "net", "tools", "crypto", "sync", "text", "time":
return atLeastGo1(branch, 13) && atLeastGo1(goBranch, 13)
}
return false
cmd/coordinator: test x/mobile on android emu builders; and exp on some Updates golang/go#23824 Change-Id: Iee70bcbfef6a271fdd26694d848a703a200dc0c0 Reviewed-on: https://go-review.googlesource.com/c/164478 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-28 20:32:45 +03:00
},
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
switch repo {
dashboard: stop testing x/build on even more builders Follow up to CL 210378 after looking at: https://build.golang.org/?repo=golang.org%2fx%2fbuild Updates golang/go#30640 Change-Id: I81505a9ca2985ffa6e75922665b14ef4de2af860 Reviewed-on: https://go-review.googlesource.com/c/build/+/211100 Reviewed-by: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-12-12 20:43:59 +03:00
case "build", "blog", "talks", "review", "tour", "website":
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
return false
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
return atLeastGo1(branch, 13) && atLeastGo1(goBranch, 13)
dashboard: add a GOOS=linux builder & trybot for Android Updates golang/go#30482 Change-Id: I198dd40dfdb86b20e681c373141754680351e82f Reviewed-on: https://go-review.googlesource.com/c/164740 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-03-01 19:54:25 +03:00
},
env/android-amd64-emu: add new Android emulator container + VM image Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:35:29 +03:00
env: []string{
"GOARCH=amd64",
"GOOS=android",
"GOHOSTARCH=amd64",
"GOHOSTOS=linux",
"CGO_ENABLED=1",
},
})
dashboard: add interim illumos builder While the work to make illumos a first class GCE guest is completed, use this interim zone provided by an illumos community member to run illumos builds. Updates golang/go#15581 Change-Id: I1784847e5407894d01ce0aadf489b38d7e5c1924 Reviewed-on: https://go-review.googlesource.com/c/build/+/201597 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-10-17 11:48:13 +03:00
addBuilder(BuildConfig{
Name: "illumos-amd64",
HostType: "host-illumos-amd64-jclulow",
MinimumGoVersion: types.MajorMinor{1, 13},
})
x/build: add Solaris amd64 release builder to dashboard Fixes golang/go#20949 Change-Id: I4f85ee76783a1240a2ebd4a2a1f167ab849fcebe Reviewed-on: https://go-review.googlesource.com/47811 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-07 23:41:20 +03:00
addBuilder(BuildConfig{
Name: "solaris-amd64-oraclerel",
HostType: "host-solaris-oracle-amd64-oraclerel",
Notes: "Oracle Solaris release version",
})
all: add ppc64 (big endian) builders Change-Id: I0fddda33179555a50b6f03dcff13338544b742c3 Reviewed-on: https://go-review.googlesource.com/30599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-06 23:30:58 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "linux-ppc64-buildlet",
HostType: "host-linux-ppc64-osu",
FlakyNet: true,
distTestAdjust: ppc64DistTestPolicy,
all: add ppc64 (big endian) builders Change-Id: I0fddda33179555a50b6f03dcff13338544b742c3 Reviewed-on: https://go-review.googlesource.com/30599 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-10-06 23:30:58 +03:00
})
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "linux-ppc64le-buildlet",
HostType: "host-linux-ppc64le-osu",
FlakyNet: true,
distTestAdjust: ppc64DistTestPolicy,
dashboard: add arm64 buildlet, update ppc64 buildlet definition Change-Id: I25c6de0f97c3338f9d32f4125b2c7c68c3f3c13c Reviewed-on: https://go-review.googlesource.com/21656 Reviewed-by: David Crawshaw <crawshaw@golang.org>
2016-04-07 22:27:15 +03:00
})
dashboard: add linux-ppc64le POWER9 builder This change adds an extra linux-ppc64le builder running on POWER9. Fixes golang/go#30138 Change-Id: I23bdfc116bb89c6d96be330c2c87827c47705b6b Reviewed-on: https://go-review.googlesource.com/c/163757 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:05:12 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "linux-ppc64le-power9osu",
HostType: "host-linux-ppc64le-power9-osu",
FlakyNet: true,
distTestAdjust: ppc64DistTestPolicy,
dashboard: add linux-ppc64le POWER9 builder This change adds an extra linux-ppc64le builder running on POWER9. Fixes golang/go#30138 Change-Id: I23bdfc116bb89c6d96be330c2c87827c47705b6b Reviewed-on: https://go-review.googlesource.com/c/163757 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-25 23:05:12 +03:00
})
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
addBuilder(BuildConfig{
Name: "linux-arm64-packet",
HostType: "host-linux-arm64-packet",
all: remove config for arm64 linaro builders They've been decomissioned for some months now. Our arm64 builders are on packet.net these days. Change-Id: Iaede189157e0298812b2971cdd63599f6096d73d Reviewed-on: https://go-review.googlesource.com/c/build/+/165781 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 08:16:49 +03:00
FlakyNet: true, // maybe not flaky, but here conservatively
build: update NetBSD from 7.0 to 7.1, disable builds, add arm64 packet builder Change-Id: I73b8d6e6ccaeca3b932238448ddcceead34105b4 Reviewed-on: https://go-review.googlesource.com/39851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-07 01:16:28 +03:00
})
dashboard: add host-linux-arm64-aws builder Add the host-linux-arm64-aws builder. The HostConfig IsVM method has been modified to support the concept that an EC2 VM can be created which runs the buildlet binary in a container. This configuration would have both a VM image and a container image set in the host configuration. For golang/go#36841 Change-Id: I092b48b1dda24d24bcd0b10dff0b2126f5f43017 Reviewed-on: https://go-review.googlesource.com/c/build/+/249118 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-18 20:32:05 +03:00
addBuilder(BuildConfig{
Name: "linux-arm64-aws",
HostType: "host-linux-arm64-aws",
})
dashboard: add linux-mipsle-mengzhuo builder Updates golang/go#31217 Change-Id: I7bda4f3ddf7dc76bdd974726747621f69a7acfa0 Reviewed-on: https://go-review.googlesource.com/c/build/+/191577 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-10 02:21:11 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
FlakyNet: true,
HostType: "host-linux-mipsle-mengzhuo",
Name: "linux-mips64le-mengzhuo",
SkipSnapshot: true,
distTestAdjust: mipsDistTestPolicy,
buildsRepo: mipsBuildsRepoPolicy,
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
env: []string{
"GOARCH=mips64le",
"GOHOSTARCH=mips64le",
},
})
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
FlakyNet: true,
HostType: "host-linux-mips64le-rtrk",
Name: "linux-mips64le-rtrk",
SkipSnapshot: true,
distTestAdjust: mipsDistTestPolicy,
buildsRepo: mipsBuildsRepoPolicy,
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
env: []string{
"GOARCH=mips64le",
"GOHOSTARCH=mips64le",
},
})
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
FlakyNet: true,
HostType: "host-linux-mips64le-rtrk",
Name: "linux-mipsle-rtrk",
SkipSnapshot: true,
distTestAdjust: mipsDistTestPolicy,
buildsRepo: mipsBuildsRepoPolicy,
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
env: []string{
"GOARCH=mipsle",
"GOHOSTARCH=mipsle",
},
})
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
FlakyNet: true,
HostType: "host-linux-mips64-rtrk",
Name: "linux-mips64-rtrk",
SkipSnapshot: true,
distTestAdjust: mipsDistTestPolicy,
buildsRepo: mipsBuildsRepoPolicy,
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
env: []string{
"GOARCH=mips64",
"GOHOSTARCH=mips64",
},
})
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
FlakyNet: true,
HostType: "host-linux-mips64-rtrk",
Name: "linux-mips-rtrk",
SkipSnapshot: true,
distTestAdjust: mipsDistTestPolicy,
buildsRepo: mipsBuildsRepoPolicy,
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
env: []string{
"GOARCH=mips",
"GOHOSTARCH=mips",
dashboard: add linux-mipsle-mengzhuo builder Updates golang/go#31217 Change-Id: I7bda4f3ddf7dc76bdd974726747621f69a7acfa0 Reviewed-on: https://go-review.googlesource.com/c/build/+/191577 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-10 02:21:11 +03:00
},
dashboard: add linux-riscv64-jsing builder (reverse) Also use this as the riscv64 slowbot, since linux-riscv64-unleashed has been MIA for quite some time. For golang/go#27532 Change-Id: I01309a66b96d4c35dbc9d1e08f55f5357b0acba0 Reviewed-on: https://go-review.googlesource.com/c/build/+/257917 Trust: Joel Sing <joel@sing.id.au> Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-09-27 21:25:41 +03:00
})
addBuilder(BuildConfig{
HostType: "host-linux-riscv64-joelsing",
Name: "linux-riscv64-jsing",
SkipSnapshot: true,
env: []string{"GO_TEST_TIMEOUT_SCALE=4"},
distTestAdjust: func(run bool, distTest string, isNormalTry bool) bool {
switch distTest {
case "api", "reboot":
return false
}
return run
},
buildsRepo: func(repo, branch, goBranch string) bool {
switch repo {
case "go", "net", "sys":
return branch == "master" && goBranch == "master"
default:
return false
}
},
dashboard: add linux-mipsle-mengzhuo builder Updates golang/go#31217 Change-Id: I7bda4f3ddf7dc76bdd974726747621f69a7acfa0 Reviewed-on: https://go-review.googlesource.com/c/build/+/191577 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-10 02:21:11 +03:00
})
dashboard, cmd/coordinator: add linux-riscv64 builder, fix bug from CL 205608 Updates golang/go#27532 Updates golang/go#30262 Change-Id: If1593ab6e5653a9dce625f25a16bf8a082796592 Reviewed-on: https://go-review.googlesource.com/c/build/+/206639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-12 09:19:21 +03:00
addBuilder(BuildConfig{
HostType: "host-linux-riscv64-unleashed",
Name: "linux-riscv64-unleashed",
SkipSnapshot: true,
dashboard: set GO_TEST_TIMEOUT_SCALE for the riscv64 builder It's slow like arm/mips/etc apparently. Change-Id: I7638f22100d3af29cdcf7852701a3d4c8c8fc1d5 Reviewed-on: https://go-review.googlesource.com/c/build/+/217758 Reviewed-by: Alexander Rakoczy <alex@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-02-04 20:04:12 +03:00
env: []string{"GO_TEST_TIMEOUT_SCALE=4"},
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
distTestAdjust: func(run bool, distTest string, isNormalTry bool) bool {
dashboard, cmd/coordinator: add linux-riscv64 builder, fix bug from CL 205608 Updates golang/go#27532 Updates golang/go#30262 Change-Id: If1593ab6e5653a9dce625f25a16bf8a082796592 Reviewed-on: https://go-review.googlesource.com/c/build/+/206639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-12 09:19:21 +03:00
switch distTest {
case "api", "reboot":
return false
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
dashboard, cmd/coordinator: add linux-riscv64 builder, fix bug from CL 205608 Updates golang/go#27532 Updates golang/go#30262 Change-Id: If1593ab6e5653a9dce625f25a16bf8a082796592 Reviewed-on: https://go-review.googlesource.com/c/build/+/206639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-12 09:19:21 +03:00
},
buildsRepo: func(repo, branch, goBranch string) bool {
switch repo {
case "go", "net", "sys":
return branch == "master" && goBranch == "master"
default:
return false
}
},
})
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
addBuilder(BuildConfig{
Name: "linux-s390x-ibm",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
HostType: "host-linux-s390x",
cmd/coordinator, dashboard: remove some trybots, shard others wider I'm aiming to have trybot runs finish in under 5 minutes. This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set. openbsd-386-gce58 is the slowest builder. It has an average speed of 722 seconds (and 95 percentile of 923 seconds) over the past week, and that's sharded over 4 machines. Too slow. It's not worth the resources to keep it as a trybot. It hasn't caught any interesting bugs. This builder will still run, but not as a pre-submit trybot. freebsd-386-gce101 is not slow, but we're removing it to shift its resources to shard other builders wider. The coordinator now supports varying the build sharding width based on whether a build is for a trybot or not. This CL defines separate numbers for each, sharding builds wider as needed for some trybots. freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down to 3 when not in try runs. linux-amd64-race gets one more machine during try runs, and one fewer in regular runs. linux-arm goes from 7 machines always, to 3 or 8, depending on whether it's a try run. openbsd-amd64-58 goes from 4 to 3 or 6. windows-amd64-gce goes from 4 to 2 or 6. windows-amd64-race goes from 4 to 2 or 6. darwin-amd64-10_11 goes from 3 to 3 or 4. I'll see how these do over the next few days and readjust as needed. Also in this CL: fix the constants for the expected duration of make.bash, which impact when we schedule the creation of test sharding helper buildlets. We were creating them too early before, wasting resources. Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f Reviewed-on: https://go-review.googlesource.com/29116 Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
numTestHelpers: 0,
dashboard: add solaris, ppc64le, s390x reverse buildlets Change-Id: I5061ae1da7884ed82786e8a042e966ccbe423e89 Reviewed-on: https://go-review.googlesource.com/20135 Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-03-01 07:49:24 +03:00
})
dashboard: add s390x cross-compile-only builder and release config Also modifies cmd/release to wire it up. Updates golang/go#16443 Change-Id: Ie80a224957f6f662d0aec0fd083e2a57e30436c4 Reviewed-on: https://go-review.googlesource.com/28172 Reviewed-by: Jessica Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 23:32:33 +03:00
addBuilder(BuildConfig{
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Name: "linux-s390x-crosscompile",
dashboard: remove misleading -kube suffixes from host types They used to run on Kubernetes (and still could) but they're just a container and we run them on COS nowadays. Change-Id: I21a6db90efc5071997c6d6dcae1a42a7c28afa3a Reviewed-on: https://go-review.googlesource.com/c/build/+/209957 Reviewed-by: Alexander Rakoczy <alex@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-12-05 00:27:28 +03:00
HostType: "host-s390x-cross",
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
Notes: "s390x cross-compile builder for releases; doesn't run tests",
CompileOnly: true,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
tryOnly: true, // but not in trybot set for now
dashboard: add s390x cross-compile-only builder and release config Also modifies cmd/release to wire it up. Updates golang/go#16443 Change-Id: Ie80a224957f6f662d0aec0fd083e2a57e30436c4 Reviewed-on: https://go-review.googlesource.com/28172 Reviewed-by: Jessica Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 23:32:33 +03:00
env: []string{
"CGO_ENABLED=1",
"GOARCH=s390x",
"GOHOSTARCH=amd64",
"CC_FOR_TARGET=s390x-linux-gnu-gcc",
},
})
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
addBuilder(BuildConfig{
Name: "linux-amd64-localdev",
HostType: "host-linux-amd64-localdev",
Notes: "for localhost development only",
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
tryOnly: true,
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support This adds an SSH server to farmer.golang.org on port 2222 that proxies SSH connections to users' gomote-created buildlet instances. For example: $ gomote create openbsd-amd64-60 user-bradfitz-openbsd-amd64-60-1 $ gomote ssh user-bradfitz-openbsd-amd64-60-1 Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts. OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. $ As before, if the coordinator process is restarted (or crashes, is evicted, etc), all gomote instances die. Not yet supported: * scp (help wanted) * not all host types are configured. most are. some will need slight config tweaks to the Docker image (e.g. adding openssh-server) Supports currently: * linux-amd64 (host type shared by 386, nacl) * linux-arm * linux-arm64 * darwin * freebsd * openbsd * plan9-386 * windows Implementation details: * the ssh server process listens on port 2222 in the coordinator (farmer.golang.org), which is behind a GKE TCP load balancer. * the ssh server library is github.com/gliderlabs/ssh * authentication is done via Github users' public keys. It's assumed that gomote user == github user. But there's a mapping in the code for known exceptions. * we can't give out access to this too widely. too many things are accessible from within the host environment if you look in the right places. Details omitted. But the Go team and other trusted gomote users can use this. * the buildlet binary has a new /connect-ssh handler that acts like a CONNECT request but instead of taking an explicit host:port, just says "give me your machine's SSH connection". The buildlet can also start sshd if needed for the environment. The /connect-ssh handler also installs the coordinator's public key. * a new buildlet client library method "ConnectSSH" hits the /connect-ssh handler and returns a net.Conn. * the coordinator's ssh.Handler is just running the OpenSSH ssh client. * because the OpenSSH ssh child process can't connect to a net.Conn, an emphemeral localhost port is created on the coordinator to proxy between the ssh client and the net.Conn returned by ConnectSSH. * The /connect-ssh handler requires http.Hijacker, which requires fully compliant net.Conn implementations as of Go 1.8. So I needed to flesh out revdial too, testing it with the golang.org/x/net/nettest package. * plan9 doesn't have an ssh server, so we use 0intro's new conterm program (drawterm without GUI support) to connect to plan9 from the coordinator ssh proxy instead of using the OpenSSH ssh client binary. * windows doesn't have an ssh server, so we enable the telnet service and the coordinator ssh proxy uses telnet instead on the backend on the private network. (There is a Windows ssh server but only in new versions.) Happy debugging over ssh! Fixes golang/go#19956 Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0 Reviewed-on: https://go-review.googlesource.com/50750 Reviewed-by: Herbie Ong <herbie@google.com> Reviewed-by: Jessie Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
})
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "dragonfly-amd64",
HostType: "host-dragonfly-amd64-master",
Notes: "DragonFly BSD master, run by DragonFly team",
distTestAdjust: noTestDirAndNoReboot,
SkipSnapshot: true,
dashboard: disable failing repos on misconfigured builders Updates golang/go#32836 Updates golang/go#31567 Updates golang/go#11811 Change-Id: I5443b61cf7732abf906ce2e93eca5408579a55c8 Reviewed-on: https://go-review.googlesource.com/c/build/+/192327 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-08-29 19:37:31 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
return atLeastGo1(goBranch, 14) && buildRepoByDefault(repo)
dashboard: disable failing repos on misconfigured builders Updates golang/go#32836 Updates golang/go#31567 Updates golang/go#11811 Change-Id: I5443b61cf7732abf906ce2e93eca5408579a55c8 Reviewed-on: https://go-review.googlesource.com/c/build/+/192327 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-08-29 19:37:31 +03:00
},
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
})
dashboard: update Dragonfly tip policy for ABI change, add release builder From https://github.com/golang/go/issues/34958#issuecomment-543852995 : > Go's DragonFly support policy is that we support the latest stable > release primarily, but also try to keep DragonFly master passing, in > prep for it to become the latest stable release. > > But that does mean we need one more builder at the moment. Updates golang/go#34958 Updates golang/go#23060 Change-Id: I84be7c64eac593dee2252c397f9529deea13605a Reviewed-on: https://go-review.googlesource.com/c/build/+/202478 Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 18:00:32 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "dragonfly-amd64-5_8",
HostType: "host-dragonfly-amd64-5_8",
Notes: "DragonFly BSD 5.8 release",
distTestAdjust: noTestDirAndNoReboot,
SkipSnapshot: true,
dashboard: update Dragonfly tip policy for ABI change, add release builder From https://github.com/golang/go/issues/34958#issuecomment-543852995 : > Go's DragonFly support policy is that we support the latest stable > release primarily, but also try to keep DragonFly master passing, in > prep for it to become the latest stable release. > > But that does mean we need one more builder at the moment. Updates golang/go#34958 Updates golang/go#23060 Change-Id: I84be7c64eac593dee2252c397f9529deea13605a Reviewed-on: https://go-review.googlesource.com/c/build/+/202478 Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-21 18:00:32 +03:00
})
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "freebsd-arm-paulzhol",
HostType: "host-freebsd-arm-paulzhol",
distTestAdjust: noTestDirAndNoReboot,
SkipSnapshot: true,
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
// This was a fragile little machine with limited memory.
// Only run a few of the core subrepos for now while
// we figure out what's killing it.
switch repo {
case "go", "sys", "net":
return true
}
return false
},
dashboard: update dragonfly-amd64 and freebsd-arm to be buildlet-based Updates golang/go#21191 Change-Id: I636204e9e4cbe77e34b2bbad65312fb6f2ace2dc Reviewed-on: https://go-review.googlesource.com/53490 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-06 01:56:18 +03:00
env: []string{
"GOARM=7",
"CGO_ENABLED=1",
},
})
dashboard: add FreeBSD arm64 builder Updates golang/go#24715 Change-Id: I8eb3f6372d69718b6f453e5415cdf16aa00ccae6 Reviewed-on: https://go-review.googlesource.com/c/build/+/198597 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-10-03 15:56:19 +03:00
addBuilder(BuildConfig{
Name: "freebsd-arm64-dmgk",
HostType: "host-freebsd-arm64-dmgk",
dashboard: only build tip on freebsd-arm64-dmgk builder Updates golang/go#24715 Change-Id: I9f1aeac0d3b5f625e0f68b6bad17445a41a87b22 Reviewed-on: https://go-review.googlesource.com/c/build/+/203420 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-25 22:21:38 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
return atLeastGo1(goBranch, 14) && buildRepoByDefault(repo)
dashboard: only build tip on freebsd-arm64-dmgk builder Updates golang/go#24715 Change-Id: I9f1aeac0d3b5f625e0f68b6bad17445a41a87b22 Reviewed-on: https://go-review.googlesource.com/c/build/+/203420 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-25 22:21:38 +03:00
},
dashboard: add FreeBSD arm64 builder Updates golang/go#24715 Change-Id: I8eb3f6372d69718b6f453e5415cdf16aa00ccae6 Reviewed-on: https://go-review.googlesource.com/c/build/+/198597 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-10-03 15:56:19 +03:00
})
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
addBuilder(BuildConfig{
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
Name: "plan9-arm",
HostType: "host-plan9-arm-0intro",
distTestAdjust: noTestDirAndNoReboot,
buildsRepo: onlyMasterDefault,
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
})
addBuilder(BuildConfig{
dashboard: use snapshots on Plan 9 builders This change enables snapshots on Plan 9 builders, so running the tests for each of the golang.org/x repositories will not require to rebuild Go every time. Change-Id: I305b713a48c155ebba2c03f39633d17a6fd71e6f Reviewed-on: https://go-review.googlesource.com/c/build/+/208359 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-22 22:19:11 +03:00
Name: "plan9-amd64-9front",
HostType: "host-plan9-amd64-0intro",
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
distTestAdjust: func(run bool, distTest string, isNormalTry bool) bool {
run = noTestDirAndNoReboot(run, distTest, isNormalTry)
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
if strings.HasPrefix(distTest, "test:") || distTest == "reboot" {
return false // skip test
dashboard: disable cmd/go tests on plan9-amd64-9front The cmd/go tests are timing out on the plan9/amd64 builder because they take more than 20m to complete. This change disables the cmd/go tests on the plan9/amd64 builder, like it was done on the plan9/386 builder. Updates golang/go#31910. Change-Id: I157b472da8498ddce0f641cccb79bc50ae65ca3b Reviewed-on: https://go-review.googlesource.com/c/build/+/175898 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-05-08 20:42:00 +03:00
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
switch distTest {
dashboard: disable cmd/go tests on plan9-amd64-9front The cmd/go tests are timing out on the plan9/amd64 builder because they take more than 20m to complete. This change disables the cmd/go tests on the plan9/amd64 builder, like it was done on the plan9/386 builder. Updates golang/go#31910. Change-Id: I157b472da8498ddce0f641cccb79bc50ae65ca3b Reviewed-on: https://go-review.googlesource.com/c/build/+/175898 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-05-08 20:42:00 +03:00
case "api",
"go_test:cmd/go": // takes over 20 minutes without working SMP
return false
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
dashboard: disable cmd/go tests on plan9-amd64-9front The cmd/go tests are timing out on the plan9/amd64 builder because they take more than 20m to complete. This change disables the cmd/go tests on the plan9/amd64 builder, like it was done on the plan9/386 builder. Updates golang/go#31910. Change-Id: I157b472da8498ddce0f641cccb79bc50ae65ca3b Reviewed-on: https://go-review.googlesource.com/c/build/+/175898 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-05-08 20:42:00 +03:00
},
dashboard: don't build x/exp and x/mobile on the plan9 builders Updates golang/go#11811 Change-Id: I776652a3b980772f34d65d8c245a143814aadb6a Reviewed-on: https://go-review.googlesource.com/c/build/+/209338 Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-28 13:33:00 +03:00
buildsRepo: onlyMasterDefault,
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
})
dashboard: add plan9-386-0intro builder This change adds the plan9-386-0intro builder configuration. Updates golang/go#29801. Change-Id: I5f64f4592a499eb1691498805fd0dfa5082c33d5 Reviewed-on: https://go-review.googlesource.com/c/build/+/175901 Run-TryBot: David du Colombier <0intro@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-09 19:52:40 +03:00
addBuilder(BuildConfig{
dashboard: use snapshots on Plan 9 builders This change enables snapshots on Plan 9 builders, so running the tests for each of the golang.org/x repositories will not require to rebuild Go every time. Change-Id: I305b713a48c155ebba2c03f39633d17a6fd71e6f Reviewed-on: https://go-review.googlesource.com/c/build/+/208359 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-22 22:19:11 +03:00
Name: "plan9-386-0intro",
HostType: "host-plan9-386-0intro",
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
distTestAdjust: func(run bool, distTest string, isNormalTry bool) bool {
run = noTestDirAndNoReboot(run, distTest, isNormalTry)
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
switch distTest {
dashboard: add plan9-386-0intro builder This change adds the plan9-386-0intro builder configuration. Updates golang/go#29801. Change-Id: I5f64f4592a499eb1691498805fd0dfa5082c33d5 Reviewed-on: https://go-review.googlesource.com/c/build/+/175901 Run-TryBot: David du Colombier <0intro@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-09 19:52:40 +03:00
case "api",
"go_test:cmd/go": // takes over 20 minutes without working SMP
return false
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
dashboard: add plan9-386-0intro builder This change adds the plan9-386-0intro builder configuration. Updates golang/go#29801. Change-Id: I5f64f4592a499eb1691498805fd0dfa5082c33d5 Reviewed-on: https://go-review.googlesource.com/c/build/+/175901 Run-TryBot: David du Colombier <0intro@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-09 19:52:40 +03:00
},
dashboard: don't build x/exp and x/mobile on the plan9 builders Updates golang/go#11811 Change-Id: I776652a3b980772f34d65d8c245a143814aadb6a Reviewed-on: https://go-review.googlesource.com/c/build/+/209338 Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-28 13:33:00 +03:00
buildsRepo: onlyMasterDefault,
dashboard: add plan9-386-0intro builder This change adds the plan9-386-0intro builder configuration. Updates golang/go#29801. Change-Id: I5f64f4592a499eb1691498805fd0dfa5082c33d5 Reviewed-on: https://go-review.googlesource.com/c/build/+/175901 Run-TryBot: David du Colombier <0intro@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-05-09 19:52:40 +03:00
})
dashboard: add aix-ppc64 builder This is a starting point for adding an aix-ppc64 reverse builder. There can be further adjustments made to it as needed. Updates golang/go#27160. Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508 Reviewed-on: https://go-review.googlesource.com/131335 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-24 21:41:05 +03:00
addBuilder(BuildConfig{
dashboard: require at least Go 1.12 for aix builder Fixes golang/go#30799 Change-Id: I0b3b53b715e2dc00f10548f74443224190003818 Reviewed-on: https://go-review.googlesource.com/c/build/+/167357 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-03-13 11:29:15 +03:00
Name: "aix-ppc64",
HostType: "host-aix-ppc64-osuosl",
MinimumGoVersion: types.MajorMinor{1, 12},
dashboard: add /opt/freeware/bin to PATH on aix builder Updates golang/go#29065 Fixes golang/go#29078 Change-Id: Ifa9355c9dc988a460b6198913431647ec2c5e6ac Reviewed-on: https://go-review.googlesource.com/c/152457 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-12-04 17:46:08 +03:00
env: []string{
"PATH=/opt/freeware/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin",
},
dashboard: don't build x/net master against Go 1.12 on aix Per request in https://golang.org/issue/31564#issuecomment-484786144 Change-Id: I52e496083b84aa021e7d7cdcda19c96980e1518f Reviewed-on: https://go-review.googlesource.com/c/build/+/172878 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-19 17:37:11 +03:00
buildsRepo: func(repo, branch, goBranch string) bool {
switch repo {
case "net":
// The x/net package wasn't working in Go 1.12; AIX folk plan to have
// it ready by Go 1.13. See https://golang.org/issue/31564#issuecomment-484786144
return atLeastGo1(branch, 13) && atLeastGo1(goBranch, 13)
dashboard: re-enable "review" repo on Dragonfly and AIX builders This partially reverts CL 192327. Reason for revert: x/review failure turned out to be a bug in the test. Updates golang/go#32836 Change-Id: I189ad4ebc1b48fbdbb64f3e42ab1459761ff0c8c Reviewed-on: https://go-review.googlesource.com/c/build/+/211497 Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-12-16 22:09:01 +03:00
case "tools", "tour", "website":
dashboard: extend aix-ppc64 skip to the website repo Updates golang/go#31567 Change-Id: Ia8d996a70166c4395393f4674af87ad755104fe1 Reviewed-on: https://go-review.googlesource.com/c/build/+/192335 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-08-29 22:05:21 +03:00
// The PATH on this builder is misconfigured in a way that causes
// any test that executes a 'go' command as a subprocess to fail.
dashboard: disable failing repos on misconfigured builders Updates golang/go#32836 Updates golang/go#31567 Updates golang/go#11811 Change-Id: I5443b61cf7732abf906ce2e93eca5408579a55c8 Reviewed-on: https://go-review.googlesource.com/c/build/+/192327 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-08-29 19:37:31 +03:00
// (https://golang.org/issue/31567).
dashboard: extend aix-ppc64 skip to the website repo Updates golang/go#31567 Change-Id: Ia8d996a70166c4395393f4674af87ad755104fe1 Reviewed-on: https://go-review.googlesource.com/c/build/+/192335 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-08-29 22:05:21 +03:00
// Skip affected repos until the builder is fixed.
dashboard: disable failing repos on misconfigured builders Updates golang/go#32836 Updates golang/go#31567 Updates golang/go#11811 Change-Id: I5443b61cf7732abf906ce2e93eca5408579a55c8 Reviewed-on: https://go-review.googlesource.com/c/build/+/192327 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-08-29 19:37:31 +03:00
return false
dashboard: don't build x/net master against Go 1.12 on aix Per request in https://golang.org/issue/31564#issuecomment-484786144 Change-Id: I52e496083b84aa021e7d7cdcda19c96980e1518f Reviewed-on: https://go-review.googlesource.com/c/build/+/172878 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-19 17:37:11 +03:00
}
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
return atLeastGo1(branch, 12) && atLeastGo1(goBranch, 12) && buildRepoByDefault(repo)
dashboard: don't build x/net master against Go 1.12 on aix Per request in https://golang.org/issue/31564#issuecomment-484786144 Change-Id: I52e496083b84aa021e7d7cdcda19c96980e1518f Reviewed-on: https://go-review.googlesource.com/c/build/+/172878 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-19 17:37:11 +03:00
},
dashboard: add aix-ppc64 builder This is a starting point for adding an aix-ppc64 reverse builder. There can be further adjustments made to it as needed. Updates golang/go#27160. Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508 Reviewed-on: https://go-review.googlesource.com/131335 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-24 21:41:05 +03:00
})
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
}
dashboard: add s390x cross-compile-only builder and release config Also modifies cmd/release to wire it up. Updates golang/go#16443 Change-Id: Ie80a224957f6f662d0aec0fd083e2a57e30436c4 Reviewed-on: https://go-review.googlesource.com/28172 Reviewed-by: Jessica Frazelle <me@jessfraz.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-08-30 23:32:33 +03:00
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
// addBuilder adds c to the Builders map after doing some sanity
// checks.
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
func addBuilder(c BuildConfig) {
if c.Name == "" {
panic("empty name")
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
if c.HostType == "" {
panic(fmt.Sprintf("missing HostType for builder %q", c.Name))
}
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
if _, dup := Builders[c.Name]; dup {
dashboard: convert plan9-arm, plan9-amd64-9front, nacl-arm to buildlets Updates golang/go#21191 Change-Id: I577dad51d490d7e281ccde9fa7d82bb1bee95bd9 Reviewed-on: https://go-review.googlesource.com/53417 Reviewed-by: David du Colombier <0intro@gmail.com>
2017-08-06 18:57:06 +03:00
panic("dup name " + c.Name)
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
}
all: split builder config into builder & host configs Our builders are named of the form "GOOS-GOARCH" or "GOOS-GOARCH-suffix". Over time we've grown many builders. This CL doesn't change that. Builders continue to be named and operate as before. Previously the build configuration file (dashboard/builders.go) made each builder type ("linux-amd64-race", etc) define how to create a host running a buildlet of that type, even though many builders had identical host configs. For example, these builders all share the same host type (a Kubernetes container): linux-amd64 linux-amd64-race linux-386 linux-386-387 And these are the same host type (a GCE VM): windows-amd64-gce windows-amd64-race windows-386-gce This CL creates a new concept of a "hostType" which defines how the buildlet is created (Kube, GCE, Reverse, and how), and then each builder itself references a host type. Users never see the hostType. (except perhaps in gomote list output) But they at least never need to care about them. Reverse buildlets now can only be one hostType at a time, which simplifies things. We were no longer using multiple roles per machine once moving to VMs for OS X. gomote continues to operate as it did previously but its underlying protocol changed and clients will need to be updated. As a new feature, gomote now has a new flag to let you reuse a buildlet host connection for different builder rules if they share the same underlying host type. But users can ignore that. This CL is a long-standing TODO (previously attempted and aborted) and will make many things easier and faster, including the linux-arm cross-compilation effort, and keeping pre-warmed buildlets of VM types ready to go. Updates golang/go#17104 Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2 Reviewed-on: https://go-review.googlesource.com/29551 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
if _, ok := Hosts[c.HostType]; !ok {
panic(fmt.Sprintf("undefined HostType %q for builder %q", c.HostType, c.Name))
cmd/coordinator: start of a Kubernetes buildlet pool Updates golang/go#12546 Change-Id: Ia70baeb1df441509f70c77e7b94e7eb9fddccbf3 Reviewed-on: https://go-review.googlesource.com/14392 Reviewed-by: Evan Brown <evanbrown@google.com>
2015-09-09 01:18:47 +03:00
}
dashboard, cmd/coordinator: clean up snapshot code, add snapshot opt-out The snapshot code had an old workaround that's no longer relevant. Remove it. Also, add a BuildConfig.SkipSnapshot bool, and use it for the slow mips builders. Fixes golang/go#19953 Change-Id: I114bb0a524184eaaae5be4715ce63f6adc519c2e Reviewed-on: https://go-review.googlesource.com/40505 Reviewed-by: Sarah Adams <shadams@google.com>
2017-04-13 07:34:42 +03:00
if c.SkipSnapshot && (c.numTestHelpers > 0 || c.numTryTestHelpers > 0) {
panic(fmt.Sprintf("config %q's SkipSnapshot is not compatible with sharded test helpers", c.Name))
}
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
types := 0
all: rename Kube to Container and IsGCE to IsVM Once containers run on COS instead of Kubernetes, one name (Kube*) is wrong and the other (GCE) is ambiguous. So rename them now to be more specific. No behavior changes. Just renaming in this step, to reduce size of next CL. Updates golang/go#25108 Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b Reviewed-on: https://go-review.googlesource.com/111639 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 19:36:05 +03:00
for _, fn := range []func() bool{c.IsReverse, c.IsContainer, c.IsVM} {
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
if fn() {
types++
}
}
if types != 1 {
all: use Container-Optimized VMs instead of Kubernetes for buildlet containers Fixes golang/go#25108 Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9 Reviewed-on: https://go-review.googlesource.com/111267 Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 05:42:17 +03:00
panic(fmt.Sprintf("build config %q host type inconsistent (must be Reverse, Image, or VM)", c.Name))
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
}
all: change dashboard.BuildConfig to be a pointer type, like HostConfig BuildConfig grew way too large and has too many slices & such to be a value type. Make it a pointer type, which matches HostConfig. Change-Id: Ie625bece9d6d8c1ec6cff26e77416bd9b1f256d8 Reviewed-on: https://go-review.googlesource.com/c/145077 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 20:52:25 +03:00
Builders[c.Name] = &c
dashboard: move builder configuration out of coordinator into a package The new package will be used by the buildlet command line tool in the future. Also use the metadata package in the coordinator for the master build key, since it was already depending on that package. Might as well use it. Change-Id: I6ad7593787ebba364090951ccbaecb4aa38e1dd6 Reviewed-on: https://go-review.googlesource.com/2920 Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-01-15 23:46:22 +03:00
}
all: misc cleanup, docs, actually add Alpine builder -- move policy of which builders are trybots out of coordinator and into dashboard/builders.go. -- move some GCE-specific code from coordinator.go to gce.go. -- rename an old "watcher" reference to "gitmirror" -- add some docs -- actually add the Alpine builder, missing from https://golang.org/cl/33890 Fixes golang/go#17891 Change-Id: Ia63671ca09aec322ed57b3663e0ac5042cdc56f2 Reviewed-on: https://go-review.googlesource.com/40395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-04-12 03:35:37 +03:00
dashboard: promote misc-compile-darwinarm64 builder to a TryBot Testing shows that misc-compile-darwinarm64 is working as expected on recent commits. Promote it to a normal misc-compile TryBot now, with the exception that it needs to run on Go 1.16 and newer only. Fixes golang/go#42341. Change-Id: Ib4de19a2ff904fb9de056db5076ac63690eb934c Reviewed-on: https://go-review.googlesource.com/c/build/+/267123 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-11-03 06:25:51 +03:00
// tryNewMiscCompile is an intermediate step towards adding a real addMiscCompile TryBot.
// It adds a post-submit-only builder with KnownIssue, GoDeps set to the provided values,
// and runs on a limited set of branches to get test results without potential disruption
// for contributors. It can be modified as needed when onboarding a misc-compile builder.
func tryNewMiscCompile(suffix, rx string, knownIssue int, goDeps []string) {
if knownIssue == 0 {
panic("tryNewMiscCompile: knownIssue parameter must be non-zero")
}
addBuilder(BuildConfig{
Name: "misc-compile" + suffix,
HostType: "host-linux-jessie",
buildsRepo: func(repo, branch, goBranch string) bool { return repo == "go" && branch == "master" },
KnownIssue: knownIssue,
GoDeps: goDeps,
env: []string{"GO_DISABLE_OUTBOUND_NETWORK=1"},
CompileOnly: true,
Notes: fmt.Sprintf("Tries buildall.bash to cross-compile & vet std+cmd packages for "+rx+", but doesn't run any tests. See golang.org/issue/%d.", knownIssue),
allScriptArgs: []string{
// Filtering pattern to buildall.bash:
rx,
},
})
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
// fasterTrybots is a distTestAdjust policy function.
// It skips (returns false) the test/ directory and reboot tests for trybots.
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
func fasterTrybots(run bool, distTest string, isNormalTry bool) bool {
if isNormalTry {
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
if strings.HasPrefix(distTest, "test:") || distTest == "reboot" {
builders: skip the reboot test in more places, especially on slower builders It's not important to run the new "reboot" bootstrapping test on all builders; a few is fine. Skip it on plan9-arm, etc, and on trybots that aren't super widely sharded. Change-Id: I35d5408eb5e034f42df5976f2b9b7f7d478d580a Reviewed-on: https://go-review.googlesource.com/c/build/+/167998 Reviewed-by: David du Colombier <0intro@gmail.com>
2019-03-18 01:28:15 +03:00
return false // skip test
}
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
// noTestDirAndNoReboot is a distTestAdjust policy function.
// It skips (returns false) the test/ directory and reboot tests for all builds.
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
func noTestDirAndNoReboot(run bool, distTest string, isNormalTry bool) bool {
builders: skip the reboot test in more places, especially on slower builders It's not important to run the new "reboot" bootstrapping test on all builders; a few is fine. Skip it on plan9-arm, etc, and on trybots that aren't super widely sharded. Change-Id: I35d5408eb5e034f42df5976f2b9b7f7d478d580a Reviewed-on: https://go-review.googlesource.com/c/build/+/167998 Reviewed-by: David du Colombier <0intro@gmail.com>
2019-03-18 01:28:15 +03:00
if strings.HasPrefix(distTest, "test:") || distTest == "reboot" {
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
return false // skip test
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
cmd/coordinator, dashboard: add mac race builder, skip redundant trybot tests This adds a darwin-amd64-race builder. Of the four platforms that support the race detector, this was the only one missing a builder. To make up for the extra work, skip some redundant tests that the Mac (and other) builds do. The test directory is really slow, and doesn't have platform-specific code. We have amd64 & 386 test coverage for that directory via the Linux builders, which is fine. No need to do it on some of the heavier VM builders, or for trybots. Fixes golang/go#17674 Change-Id: I888119b3fdd28884b87a59d78746ab1fef6960c7 Reviewed-on: https://go-review.googlesource.com/82395 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 03:40:40 +03:00
}
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
// ppc64DistTestPolicy is a distTestAdjust policy function
dashboard: skip reboot test on ppc64x builders Updates golang/go#35233 Change-Id: Icfa19da5d0dd3c47ffa94d11f069b8836c86cdcc Reviewed-on: https://go-review.googlesource.com/c/build/+/204038 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 18:02:19 +03:00
// that's shared by linux-ppc64le, -ppc64le-power9osu, and -ppc64.
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
func ppc64DistTestPolicy(run bool, distTest string, isNormalTry bool) bool {
dashboard: skip reboot test on ppc64x builders Updates golang/go#35233 Change-Id: Icfa19da5d0dd3c47ffa94d11f069b8836c86cdcc Reviewed-on: https://go-review.googlesource.com/c/build/+/204038 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 18:02:19 +03:00
if distTest == "reboot" {
// Skip test. It seems to use a lot of memory?
// See https://golang.org/issue/35233.
return false
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
dashboard: skip reboot test on ppc64x builders Updates golang/go#35233 Change-Id: Icfa19da5d0dd3c47ffa94d11f069b8836c86cdcc Reviewed-on: https://go-review.googlesource.com/c/build/+/204038 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 18:02:19 +03:00
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
// mipsDistTestPolicy is a distTestAdjust policy function
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
// that's shared by the slow mips builders.
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
func mipsDistTestPolicy(run bool, distTest string, isNormalTry bool) bool {
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
switch distTest {
case "api", "reboot":
return false
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
dashboard: add MIPS builders at RT-RK Updates golang/go#31217 Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533 Reviewed-on: https://go-review.googlesource.com/c/build/+/204043 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-29 23:35:07 +03:00
}
// mipsBuildsRepoPolicy is a buildsRepo policy function
// that's shared by the slow mips builders.
func mipsBuildsRepoPolicy(repo, branch, goBranch string) bool {
switch repo {
case "go", "net", "sys":
return branch == "master" && goBranch == "master"
default:
return false
}
}
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
// TryBuildersForProject returns the builders that should run as part of
// a TryBot set for the given project.
// The project argument is of the form "go", "net", "sys", etc.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// The branch is the branch of that project ("master", "release-branch.go1.12", etc)
// The goBranch is the branch of Go to use. If proj == "go", then branch == goBranch.
func TryBuildersForProject(proj, branch, goBranch string) []*BuildConfig {
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
var confs []*BuildConfig
for _, conf := range Builders {
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
if conf.BuildsRepoTryBot(proj, branch, goBranch) {
dashboard, cmd/coordinator: move more config out of coordinator into dashboard And with the increased flexibility, enable trybots for more repos for the "sys" project. Details: https://github.com/golang/go/issues/24893#issuecomment-433517519 Fixes golang/go#24893 Change-Id: I89ce6f219be986201332f8fad11c774bdd6246da Reviewed-on: https://go-review.googlesource.com/c/145157 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-26 22:21:58 +03:00
confs = append(confs, conf)
}
}
sort.Slice(confs, func(i, j int) bool {
return confs[i].Name < confs[j].Name
})
return confs
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
dashboard: treat dev branches as current If a dev branch is active, it will be current. If it is not active, it is probably not current, but we don't build on the branch so it doesn't matter anyway. Specifically, the dev.link branch is currently under active development, and it would be good to have a large set of builders run on the branch to catch problems. (E.g., AIX build was broken but not noticed.) Fixes golang/go#37953. Change-Id: Idd7e3edaaff1b4d2cee17da552a96df2b00716e9 Reviewed-on: https://go-review.googlesource.com/c/build/+/223381 Run-TryBot: Cherry Zhang <cherryyz@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-03-14 00:09:21 +03:00
// atLeastGo1 reports whether branch is "release-branch.go1.N" where N >= min.
// It assumes "master" and "dev.*" branches are already greater than min, and
// always includes them.
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
func atLeastGo1(branch string, min int) bool {
if branch == "master" {
return true
}
dashboard: treat dev branches as current If a dev branch is active, it will be current. If it is not active, it is probably not current, but we don't build on the branch so it doesn't matter anyway. Specifically, the dev.link branch is currently under active development, and it would be good to have a large set of builders run on the branch to catch problems. (E.g., AIX build was broken but not noticed.) Fixes golang/go#37953. Change-Id: Idd7e3edaaff1b4d2cee17da552a96df2b00716e9 Reviewed-on: https://go-review.googlesource.com/c/build/+/223381 Run-TryBot: Cherry Zhang <cherryyz@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-03-14 00:09:21 +03:00
if strings.HasPrefix(branch, "dev.") {
// Treat dev branches current.
// If a dev branch is active, it will be current.
// If it is not active, it doesn't matter anyway.
// TODO: dev.boringcrypto.go1.N branches may be the
// exception. Currently we only build boringcrypto
// on linux/amd64 and windows/386, which support all
// versions of Go, so it doesn't actually matter.
return true
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
major, minor, ok := version.ParseReleaseBranch(branch)
return ok && major == 1 && minor >= min
}
dashboard: disable darwin-amd64-10_11 builder on Go 1.15+ Go 1.15 will require macOS 10.12 and newer¹, so keep the OS X 10.11 builder enabled only on release branches for 1.14 and older. ¹ https://golang.org/doc/go1.14#darwin Fixes golang/go#37425 Change-Id: I89ab0eb56729fe16fb3a524c8f7b9bb25751d708 Reviewed-on: https://go-review.googlesource.com/c/build/+/222797 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
2020-03-10 20:26:20 +03:00
// atMostGo1 reports whether branch is "release-branch.go1.N" where N <= max.
// It assumes "master" branch is already greater than max, and doesn't include it.
func atMostGo1(branch string, max int) bool {
major, minor, ok := version.ParseReleaseBranch(branch)
return ok && major == 1 && minor <= max
}
dashboard: unify post-submit & trybot policy in builders.go, not coordinator This removes a bunch of TODOs about scattered policy and moves it all into builders.go. While here, * add a bunch of tests * unexport some things * rename some things * document some things * adjust FreeBSD policy as function of branch (per Go 1.12 being last to support FreeBSD 10.x, and to unblock CL 165801) * set GO_BUILDER_SET_GOPROXY=coordinator for reverse buildlets, which is necessary to remove the oauth2 & build special cases in the config * change Elias Naur's mobile builder to how I think he wants it (he was fighting the old system) * add $HOME on the Solaris smartos builder, which was missing & causing tests to fail lately * make the (currently failing) longtest builder have GOPROXY set * remove an allocation in version.ParseReleaseBranch Fixes golang/go#9603 Updates golang/go#14594 Change-Id: I50a23ad7cdf478c95b14bee9b3931ba361baacfa Reviewed-on: https://go-review.googlesource.com/c/build/+/166218 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-07 20:44:41 +03:00
// onlyGo is a common buildsRepo policy value that only builds the main "go" repo.
func onlyGo(repo, branch, goBranch string) bool { return repo == "go" }
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
dashboard: don't build x/exp and x/mobile on the plan9 builders Updates golang/go#11811 Change-Id: I776652a3b980772f34d65d8c245a143814aadb6a Reviewed-on: https://go-review.googlesource.com/c/build/+/209338 Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-28 13:33:00 +03:00
// onlyMasterDefault is a common buildsRepo policy value that only builds
// default repos on the master branch.
func onlyMasterDefault(repo, branch, goBranch string) bool {
dashboard: document and clarify defaultBuildsRepoPolicy The defaultBuildsRepoPolicy function was very important in that any changes to it would affect many builders, but it was not documented well. This refactor attempts to improve the situation. It turned out better to also rename defaultBuildsRepoPolicy to buildRepoByDefault and limit its scope to report whether a given repo should be built by default. The branch and goBranch parameters were never used, and thus misleading. If they're needed in the future, we can easily re-add them. The name defaultBuildsRepoPolicy made it sound as if it was the entire "builds repo" policy, bun reality most of the policy is implemented in the BuildConfig.buildsRepoAtAll method and is non-configurable. Only a small part of the policy was in defaultBuildsRepoPolicy. For golang/go#39156. Change-Id: Ie7c6a68bd624056362dc7d964bd8d640f665b35a Reviewed-on: https://go-review.googlesource.com/c/build/+/240687 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-07-01 21:45:10 +03:00
return branch == "master" && goBranch == "master" && buildRepoByDefault(repo)
dashboard, cmd/coordinator: remove Joyent builders Joyent.com is shutting down their public cloud, so we no longer have our GOOS=solaris or GOOS=illumos builders there. Maybe somebody will find a new place to run them. Or maybe the ports will be abandoned. We'll see. Updates golang/go#15581 Change-Id: I0590227ce61b6b298b6aa4554e5e3bc9e4c464b5 Reviewed-on: https://go-review.googlesource.com/c/build/+/200219 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-10-10 07:17:45 +03:00
}
dashboard: only test plan9 at master The plan9 port doesn't need to be tested against older Go releases. Its userbase runs at master. Change-Id: I710d98560b234c7555c7be526aa2efdb6473c439 Reviewed-on: https://go-review.googlesource.com/c/build/+/170519 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-03 08:03:37 +03:00
all: add a macOS 10.14 (Mojave) builder Fixes golang/go#27806 Change-Id: I576f563acb2c50cd0456cd4e6a1271b9aa59c9df Reviewed-on: https://go-review.googlesource.com/c/build/+/169498 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-27 02:04:31 +03:00
// disabledBuilder is a buildsRepo policy function that always return false.
func disabledBuilder(repo, branch, goBranch string) bool { return false }
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
// macTestPolicy is the test policy for Macs.
//
// We have limited Mac resources. It's not worth wasting time testing
// portable things on them. That is, if there's a slow test that will
// still fail slowly on another builder where we have more resources
// (like linux-amd64), then there's no point testing it redundantly on
// the Macs.
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
func macTestPolicy(run bool, distTest string, isNormalTry bool) bool {
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
if strings.HasPrefix(distTest, "test:") {
return false
}
switch distTest {
case "reboot", "api", "doc_progs",
"wiki", "bench_go1", "codewalk":
return false
}
cmd/coordinator, dashboard: don't skip cmd/dist tests for SlowBots When a user explicitly requests a SlowBot builder via the TRY= syntax, it is a signal that they're willing to wait longer for test results. It's more useful to not skip any tests and give the user confidence that the SlowBot run will catch all issues that would otherwise be caught only by post-submit builders after the CL is submitted. This is done by considering a TryBot that isn't an explicit SlowBot to be a "normal TryBot", and the ShouldRunDistTest policy function is provided a parameter specifying whether it's a normal TryBot (non-SlowBot) run. Tests are only skipped during normal TryBot runs, not SlowBot runs. Fixes golang/go#38279. For golang/go#34501. Change-Id: Ia11fd51c71f9de1089b6b6f4c027014893f224b9 Reviewed-on: https://go-review.googlesource.com/c/build/+/227779 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 23:24:30 +03:00
if isNormalTry {
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
switch distTest {
case "runtime:cpu124", "race", "moved_goroot":
return false
}
// TODO: more. Look at bigquery results once we have more data.
}
dashboard: refactor mechanism for adjusting cmd/dist test policy Previously, the behavior of ShouldRunDistTest could be overridden by an individual builder by setting a the shouldRunDistTest field to a custom policy implementation. A pitfall of this approach is that it requires each custom policy to reimplement the default cmd/dist test policy as well. If that isn't done when attempting to make a small adjustment to the default policy, the result is that the default policy is ignored. The fasterTrybots policy, as the name suggests, exists to skip some tests in trybot mode. Due to the problem above, it was causing some builders to run more tests in trybot mode than they would if it weren't applied. For example, the freebsd-amd64-12_0 builder should have been skipping the "api" test in trybot mode, but wasn't. This refactor fixes that by modifying the approach for how individual builders can customize the cmd/dist test behavior. Now, a function distTestAdjust is called with the first parameter specifying the default policy decision. That function can adjust the decision as it sees fit, but otherwise it preserves its initial value in situations where no change is intended to be made. For golang/go#38279. Change-Id: I220248f51abfc8e77195ae0c9134a17f7cac9bc2 Reviewed-on: https://go-review.googlesource.com/c/build/+/227777 Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-04-09 19:47:03 +03:00
return run
dashboard: skip some slow tests on Macs We have limited Mac resources. It's not worth wasting time testing portable things on them. Change-Id: Iaaa1d02582da7f8435217fbb32dcb7235c560a41 Reviewed-on: https://go-review.googlesource.com/c/build/+/169681 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-03-28 02:03:50 +03:00
}