2015-01-15 23:46:22 +03:00
|
|
|
// Copyright 2015 The Go Authors. All rights reserved.
|
|
|
|
|
// Use of this source code is governed by a BSD-style
|
|
|
|
|
// license that can be found in the LICENSE file.
|
|
|
|
|
|
|
|
|
|
// Package dashboard contains shared configuration and logic used by various
|
|
|
|
|
// pieces of the Go continuous build system.
|
|
|
|
|
package dashboard
|
|
|
|
|
|
2015-06-05 04:25:50 +03:00
|
|
|
import (
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
"fmt"
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
"os"
|
2017-04-12 03:35:37 +03:00
|
|
|
"sort"
|
2015-06-05 04:25:50 +03:00
|
|
|
"strconv"
|
|
|
|
|
"strings"
|
2016-02-15 02:59:59 +03:00
|
|
|
|
|
|
|
|
"golang.org/x/build/buildenv"
|
2015-06-05 04:25:50 +03:00
|
|
|
)
|
2015-01-15 23:46:22 +03:00
|
|
|
|
|
|
|
|
// Builders are the different build configurations.
|
|
|
|
|
// The keys are like "darwin-amd64" or "linux-386-387".
|
|
|
|
|
// This map should not be modified by other packages.
|
2017-04-12 03:35:37 +03:00
|
|
|
// Initialization happens below, via calls to addBuilder.
|
2018-10-26 20:52:25 +03:00
|
|
|
var Builders = map[string]*BuildConfig{}
|
2015-01-15 23:46:22 +03:00
|
|
|
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
// Hosts contains the names and configs of all the types of
|
|
|
|
|
// buildlets. They can be VMs, containers, or dedicated machines.
|
|
|
|
|
var Hosts = map[string]*HostConfig{
|
2018-05-14 20:22:45 +03:00
|
|
|
"host-linux-jessie": &HostConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Notes: "Debian Jessie, our standard Linux container image.",
|
2018-05-14 20:22:45 +03:00
|
|
|
ContainerImage: "linux-x86-jessie:latest",
|
|
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
|
|
|
|
|
SSHUsername: "root",
|
|
|
|
|
},
|
|
|
|
|
"host-linux-stretch": &HostConfig{
|
|
|
|
|
Notes: "Debian Stretch",
|
|
|
|
|
ContainerImage: "linux-x86-stretch:latest",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
SSHUsername: "root",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
},
|
2016-09-23 23:44:43 +03:00
|
|
|
"host-linux-armhf-cross": &HostConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Notes: "Debian Jessie with armhf cross-compiler, built from env/crosscompile/linux-armhf-jessie",
|
2018-05-05 19:36:05 +03:00
|
|
|
ContainerImage: "linux-armhf-jessie:latest",
|
2016-09-23 23:44:43 +03:00
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
|
|
|
|
|
},
|
2017-03-23 18:46:39 +03:00
|
|
|
"host-linux-armel-cross": &HostConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Notes: "Debian Jessie with armel cross-compiler, from env/crosscompile/linux-armel-stretch",
|
2018-05-05 19:36:05 +03:00
|
|
|
ContainerImage: "linux-armel-stretch:latest",
|
2017-03-23 18:46:39 +03:00
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
|
|
|
|
|
},
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
"host-linux-amd64-localdev": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 0,
|
|
|
|
|
Notes: "for localhost development of buildlets/gomote/coordinator only",
|
|
|
|
|
SSHUsername: os.Getenv("USER"),
|
|
|
|
|
},
|
2017-08-06 18:57:06 +03:00
|
|
|
"host-nacl-arm-davecheney": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
|
|
|
|
Notes: "Raspberry Pi 3",
|
|
|
|
|
OwnerGithub: "davecheney",
|
|
|
|
|
},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
"host-nacl-kube": &HostConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Notes: "Container with Native Client binaries.",
|
2018-05-05 19:36:05 +03:00
|
|
|
ContainerImage: "linux-x86-nacl:latest",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
|
|
|
|
|
},
|
2018-05-11 08:54:32 +03:00
|
|
|
"host-js-wasm": &HostConfig{
|
|
|
|
|
Notes: "Container with node.js for testing js/wasm.",
|
|
|
|
|
ContainerImage: "js-wasm:latest",
|
|
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
|
|
|
|
|
},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
"host-s390x-cross-kube": &HostConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Notes: "Container with s390x cross-compiler.",
|
2018-05-05 19:36:05 +03:00
|
|
|
ContainerImage: "linux-s390x-stretch:latest",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
|
|
|
|
|
},
|
2016-12-02 22:14:12 +03:00
|
|
|
"host-linux-x86-alpine": &HostConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Notes: "Alpine container",
|
2018-05-05 19:36:05 +03:00
|
|
|
ContainerImage: "linux-x86-alpine:latest",
|
2016-12-02 22:14:12 +03:00
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64-static",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/usr/lib/go"},
|
|
|
|
|
},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
"host-linux-clang": &HostConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Notes: "Container with clang.",
|
2018-05-05 19:36:05 +03:00
|
|
|
ContainerImage: "linux-x86-clang:latest",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
|
|
|
|
|
},
|
|
|
|
|
"host-linux-sid": &HostConfig{
|
2017-04-12 04:54:09 +03:00
|
|
|
Notes: "Debian sid, updated occasionally.",
|
2018-05-05 19:36:05 +03:00
|
|
|
ContainerImage: "linux-x86-sid:latest",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.linux-amd64",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/go1.4"},
|
|
|
|
|
},
|
2017-05-22 23:17:48 +03:00
|
|
|
"host-linux-arm-scaleway": &HostConfig{
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
IsReverse: true,
|
|
|
|
|
HermeticReverse: true,
|
|
|
|
|
ExpectNum: 50,
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
|
|
|
|
|
ReverseAliases: []string{"linux-arm", "linux-arm-arm5"},
|
|
|
|
|
SSHUsername: "root",
|
2017-05-22 23:17:48 +03:00
|
|
|
},
|
2017-03-22 23:53:27 +03:00
|
|
|
"host-linux-arm5spacemonkey": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 3,
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
|
|
|
|
|
ReverseAliases: []string{"linux-arm-arm5spacemonkey"},
|
2017-04-25 02:07:45 +03:00
|
|
|
OwnerGithub: "zeebo",
|
2017-03-22 23:53:27 +03:00
|
|
|
},
|
2016-12-10 22:10:49 +03:00
|
|
|
"host-openbsd-amd64-60": &HostConfig{
|
|
|
|
|
VMImage: "openbsd-amd64-60",
|
|
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-60.tar.gz",
|
|
|
|
|
Notes: "OpenBSD 6.0; GCE VM is built from script in build/env/openbsd-amd64",
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
SSHUsername: "gopher",
|
2016-12-10 22:10:49 +03:00
|
|
|
},
|
2016-12-15 01:41:25 +03:00
|
|
|
"host-openbsd-386-60": &HostConfig{
|
|
|
|
|
VMImage: "openbsd-386-60",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386",
|
2016-12-15 01:41:25 +03:00
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-60.tar.gz",
|
|
|
|
|
Notes: "OpenBSD 6.0; GCE VM is built from script in build/env/openbsd-386",
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
SSHUsername: "gopher",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
},
|
2017-11-24 21:14:33 +03:00
|
|
|
"host-openbsd-amd64-62": &HostConfig{
|
|
|
|
|
VMImage: "openbsd-amd64-62",
|
|
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-60.tar.gz",
|
|
|
|
|
Notes: "OpenBSD 6.2; GCE VM is built from script in build/env/openbsd-amd64",
|
|
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
},
|
2017-12-05 08:22:36 +03:00
|
|
|
"host-openbsd-386-62": &HostConfig{
|
|
|
|
|
VMImage: "openbsd-386-62-a",
|
|
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-60.tar.gz",
|
|
|
|
|
Notes: "OpenBSD 6.2; GCE VM is built from script in build/env/openbsd-386",
|
|
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
},
|
2018-10-05 01:14:50 +03:00
|
|
|
"host-openbsd-amd64-64": &HostConfig{
|
env/openbsd-amd64: enable SMT on OpenBSD 6.4
Disable Spectre/Meltdown mitigations and enable simultaneous
multithreading (SMT) on the OpenBSD 6.4 builder image.
This was done by setting hw.smt=1 in /etc/sysctl.conf when building
the image.
I considered an alternative approach of doing this at runtime via
a cmd/buildlet special case for OpenBSD. It would've likely been
faster to implement if I had the idea at the beginning. However,
by the time I saw it, I had already started to build a new OpenBSD 6.4
image with the /etc/sysctl.conf file, then tested it via debugnewvm.
At this point, there's no longer a time saving advantage, so I decided
to prefer the v2 image because it keeps OpenBSD-specific configuration
more contained in env/openbsd-* directory, rather than spreading it out
between there and the cmd/buildlet runtime code. It seems to be a
slightly better option.
The times to do a full build, as measured via cmd/debugnewvm, were:
6.2 = 19m25.218s
6.4 = 28m49.565s
6.4 with hw.smt=1 = 22m55.909s
That should translate to faster trybot runs, which is important
for open source project health.
The 386 environment doesn't need to be updated because it doesn't
support hw.smt:
sysctl: hw.smt: value is not available
Attempting to set it anyway should be harmless and okay to do,
in case it happens to get supported in the future.
We're only using amd64 environment for trybots, so it's okay
for the purposes of golang/go#28403.
Reference: https://man.openbsd.org/sysctl.2#HW_SMT_2
Fixes golang/go#28403
Change-Id: I7bc4cbf83ccbdb3aa9dd19eeabd88feb1c425811
Reviewed-on: https://go-review.googlesource.com/c/145022
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 20:05:59 +03:00
|
|
|
VMImage: "openbsd-amd64-64-v2",
|
2018-10-05 01:14:50 +03:00
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-amd64-64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-amd64-64.tar.gz",
|
env/openbsd-amd64: enable SMT on OpenBSD 6.4
Disable Spectre/Meltdown mitigations and enable simultaneous
multithreading (SMT) on the OpenBSD 6.4 builder image.
This was done by setting hw.smt=1 in /etc/sysctl.conf when building
the image.
I considered an alternative approach of doing this at runtime via
a cmd/buildlet special case for OpenBSD. It would've likely been
faster to implement if I had the idea at the beginning. However,
by the time I saw it, I had already started to build a new OpenBSD 6.4
image with the /etc/sysctl.conf file, then tested it via debugnewvm.
At this point, there's no longer a time saving advantage, so I decided
to prefer the v2 image because it keeps OpenBSD-specific configuration
more contained in env/openbsd-* directory, rather than spreading it out
between there and the cmd/buildlet runtime code. It seems to be a
slightly better option.
The times to do a full build, as measured via cmd/debugnewvm, were:
6.2 = 19m25.218s
6.4 = 28m49.565s
6.4 with hw.smt=1 = 22m55.909s
That should translate to faster trybot runs, which is important
for open source project health.
The 386 environment doesn't need to be updated because it doesn't
support hw.smt:
sysctl: hw.smt: value is not available
Attempting to set it anyway should be harmless and okay to do,
in case it happens to get supported in the future.
We're only using amd64 environment for trybots, so it's okay
for the purposes of golang/go#28403.
Reference: https://man.openbsd.org/sysctl.2#HW_SMT_2
Fixes golang/go#28403
Change-Id: I7bc4cbf83ccbdb3aa9dd19eeabd88feb1c425811
Reviewed-on: https://go-review.googlesource.com/c/145022
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-26 20:05:59 +03:00
|
|
|
Notes: "OpenBSD 6.4 with hw.smt=1; GCE VM is built from script in build/env/openbsd-amd64",
|
2018-10-05 01:14:50 +03:00
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
},
|
|
|
|
|
"host-openbsd-386-64": &HostConfig{
|
2018-10-19 22:50:28 +03:00
|
|
|
VMImage: "openbsd-386-64",
|
2018-10-05 01:14:50 +03:00
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.openbsd-386-64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-openbsd-386-64.tar.gz",
|
2018-10-19 22:50:28 +03:00
|
|
|
Notes: "OpenBSD 6.4; GCE VM is built from script in build/env/openbsd-386",
|
2018-10-05 01:14:50 +03:00
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
"host-freebsd-93-gce": &HostConfig{
|
|
|
|
|
VMImage: "freebsd-amd64-gce93",
|
|
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
SSHUsername: "gopher",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
},
|
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1.
freebsd.org says that the current supported releases are:
> Production: 11.1, 11.0, 10.4, 10.3
Previously we had builders for 9.3, 10.1, and 11.0.
We'll keep FreeBSD 9.3 for now (as it's still supported by the last
two released Go versions), but we'll stop testing FreeBSD 9.3 at tip
shortly here (that will be a separate change).
This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3,
spanning the range from the old to newest supported release. We'll
stop testing 10.4 and 11.0, as that's a bit overkill.
Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9
release notes (https://golang.org/doc/go1.9#freebsd) said that future
Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test
FreeBSD 10.3, being the oldest. (Go has historically "supported"
FreeBSD releases a bit longer than they're officially supported by
FreeBSD, partly due to our out-of-sync release cycles.)
Fixes golang/go#19303
Fixes golang/go#22854
Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f
Reviewed-on: https://go-review.googlesource.com/80297
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
|
|
|
"host-freebsd-10_3": &HostConfig{
|
2017-11-29 01:07:29 +03:00
|
|
|
VMImage: "freebsd-amd64-103-b",
|
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1.
freebsd.org says that the current supported releases are:
> Production: 11.1, 11.0, 10.4, 10.3
Previously we had builders for 9.3, 10.1, and 11.0.
We'll keep FreeBSD 9.3 for now (as it's still supported by the last
two released Go versions), but we'll stop testing FreeBSD 9.3 at tip
shortly here (that will be a separate change).
This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3,
spanning the range from the old to newest supported release. We'll
stop testing 10.4 and 11.0, as that's a bit overkill.
Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9
release notes (https://golang.org/doc/go1.9#freebsd) said that future
Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test
FreeBSD 10.3, being the oldest. (Go has historically "supported"
FreeBSD releases a bit longer than they're officially supported by
FreeBSD, partly due to our out-of-sync release cycles.)
Fixes golang/go#19303
Fixes golang/go#22854
Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f
Reviewed-on: https://go-review.googlesource.com/80297
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
|
|
|
Notes: "FreeBSD 10.3; GCE VM is built from script in build/env/freebsd-amd64",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64", // TODO(bradfitz): why was this http instead of https?
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
|
|
|
|
|
env: []string{"CC=clang"},
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
SSHUsername: "gopher",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
},
|
2018-09-28 02:43:25 +03:00
|
|
|
"host-freebsd-10_4": &HostConfig{
|
|
|
|
|
VMImage: "freebsd-amd64-104",
|
|
|
|
|
Notes: "FreeBSD 10.4; GCE VM is built from script in build/env/freebsd-amd64",
|
|
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
|
|
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
},
|
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1.
freebsd.org says that the current supported releases are:
> Production: 11.1, 11.0, 10.4, 10.3
Previously we had builders for 9.3, 10.1, and 11.0.
We'll keep FreeBSD 9.3 for now (as it's still supported by the last
two released Go versions), but we'll stop testing FreeBSD 9.3 at tip
shortly here (that will be a separate change).
This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3,
spanning the range from the old to newest supported release. We'll
stop testing 10.4 and 11.0, as that's a bit overkill.
Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9
release notes (https://golang.org/doc/go1.9#freebsd) said that future
Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test
FreeBSD 10.3, being the oldest. (Go has historically "supported"
FreeBSD releases a bit longer than they're officially supported by
FreeBSD, partly due to our out-of-sync release cycles.)
Fixes golang/go#19303
Fixes golang/go#22854
Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f
Reviewed-on: https://go-review.googlesource.com/80297
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
|
|
|
"host-freebsd-11_1": &HostConfig{
|
2017-11-29 01:07:29 +03:00
|
|
|
VMImage: "freebsd-amd64-111-b",
|
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1.
freebsd.org says that the current supported releases are:
> Production: 11.1, 11.0, 10.4, 10.3
Previously we had builders for 9.3, 10.1, and 11.0.
We'll keep FreeBSD 9.3 for now (as it's still supported by the last
two released Go versions), but we'll stop testing FreeBSD 9.3 at tip
shortly here (that will be a separate change).
This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3,
spanning the range from the old to newest supported release. We'll
stop testing 10.4 and 11.0, as that's a bit overkill.
Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9
release notes (https://golang.org/doc/go1.9#freebsd) said that future
Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test
FreeBSD 10.3, being the oldest. (Go has historically "supported"
FreeBSD releases a bit longer than they're officially supported by
FreeBSD, partly due to our out-of-sync release cycles.)
Fixes golang/go#19303
Fixes golang/go#22854
Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f
Reviewed-on: https://go-review.googlesource.com/80297
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
|
|
|
Notes: "FreeBSD 11.1; GCE VM is built from script in build/env/freebsd-amd64",
|
2017-02-27 19:05:25 +03:00
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64", // TODO(bradfitz): why was this http instead of https?
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
|
|
|
|
|
env: []string{"CC=clang"},
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
SSHUsername: "gopher",
|
2017-02-27 19:05:25 +03:00
|
|
|
},
|
2018-09-28 02:43:25 +03:00
|
|
|
"host-freebsd-11_2": &HostConfig{
|
|
|
|
|
VMImage: "freebsd-amd64-112",
|
|
|
|
|
Notes: "FreeBSD 11.2; GCE VM is built from script in build/env/freebsd-amd64",
|
|
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
|
|
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
},
|
2018-10-05 23:57:29 +03:00
|
|
|
"host-freebsd-12_0": &HostConfig{
|
|
|
|
|
VMImage: "freebsd-amd64-120a8",
|
|
|
|
|
Notes: "FreeBSD 12.0alpha8; GCE VM is built from script in build/env/freebsd-amd64",
|
|
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "https://storage.googleapis.com/$BUCKET/buildlet.freebsd-amd64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-freebsd-amd64.tar.gz",
|
|
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
},
|
2018-04-28 01:47:56 +03:00
|
|
|
"host-netbsd-amd64-8_0": &HostConfig{
|
|
|
|
|
VMImage: "netbsd-amd64-8-0-2018q1",
|
|
|
|
|
Notes: "NetBSD 8.0RC1; GCE VM is built from script in build/env/netbsd-amd64",
|
2017-11-30 05:32:26 +03:00
|
|
|
machineType: "n1-highcpu-4",
|
2017-04-07 01:16:28 +03:00
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.netbsd-amd64",
|
2017-11-30 05:32:26 +03:00
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-netbsd-amd64-2da6b33.tar.gz",
|
2017-12-01 09:55:37 +03:00
|
|
|
SSHUsername: "root",
|
2017-04-07 01:16:28 +03:00
|
|
|
},
|
2018-04-28 01:47:56 +03:00
|
|
|
// Note: the netbsd-386 host hangs during the ../test phase of all.bash,
|
|
|
|
|
// so we don't use this for now. (See the netbsd-386-8 BuildConfig below.)
|
|
|
|
|
"host-netbsd-386-8_0": &HostConfig{
|
|
|
|
|
VMImage: "netbsd-386-8-0-2018q1",
|
|
|
|
|
Notes: "NetBSD 8.0RC1; GCE VM is built from script in build/env/netbsd-386",
|
2017-11-30 05:32:26 +03:00
|
|
|
machineType: "n1-highcpu-4",
|
2017-11-29 03:44:46 +03:00
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.netbsd-386",
|
2017-12-06 02:39:30 +03:00
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-netbsd-386-0b3b511.tar.gz",
|
2018-04-28 01:47:56 +03:00
|
|
|
SSHUsername: "root",
|
2017-11-29 03:44:46 +03:00
|
|
|
},
|
2017-08-06 01:56:18 +03:00
|
|
|
"host-dragonfly-amd64-tdfbsd": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
2017-08-06 02:14:51 +03:00
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go"},
|
2017-08-06 01:56:18 +03:00
|
|
|
ReverseAliases: []string{"dragonfly-amd64"},
|
|
|
|
|
OwnerGithub: "tdfbsd",
|
|
|
|
|
},
|
|
|
|
|
"host-freebsd-arm-paulzhol": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
2017-08-06 06:57:03 +03:00
|
|
|
Notes: "Cubiboard2 1Gb RAM dual-core Cortex-A7 (Allwinner A20), FreeBSD 11.1-RELEASE",
|
2017-08-06 02:14:51 +03:00
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/usr/home/paulzhol/go1.4"},
|
2017-08-06 01:56:18 +03:00
|
|
|
ReverseAliases: []string{"freebsd-arm-paulzhol"},
|
|
|
|
|
OwnerGithub: "paulzhol",
|
|
|
|
|
},
|
2017-08-06 18:57:06 +03:00
|
|
|
"host-plan9-arm-0intro": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
|
|
|
|
Notes: "Raspberry Pi 3 Model B, Plan 9 from Bell Labs",
|
|
|
|
|
OwnerGithub: "0intro",
|
|
|
|
|
},
|
|
|
|
|
"host-plan9-amd64-0intro": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
|
|
|
|
OwnerGithub: "0intro",
|
|
|
|
|
},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
"host-plan9-386-gce": &HostConfig{
|
2018-08-21 03:01:51 +03:00
|
|
|
VMImage: "plan9-386-v6",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Notes: "Plan 9 from 0intro; GCE VM is built from script in build/env/plan9-386",
|
|
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.plan9-386",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/gobootstrap-plan9-386.tar.gz",
|
2015-01-15 23:46:22 +03:00
|
|
|
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
// We *were* using n1-standard-1 because Plan 9 can only
|
|
|
|
|
// reliably use a single CPU. Using 2 or 4 and we see
|
|
|
|
|
// test failures. See:
|
|
|
|
|
// https://golang.org/issue/8393
|
|
|
|
|
// https://golang.org/issue/9491
|
|
|
|
|
// n1-standard-1 has 3.6 GB of memory which WAS (see below)
|
|
|
|
|
// overkill (userspace probably only sees 2GB anyway),
|
|
|
|
|
// but it's the cheapest option. And plenty to keep
|
|
|
|
|
// our ~250 MB of inputs+outputs in its ramfs.
|
|
|
|
|
//
|
|
|
|
|
// But the docs says "For the n1 series of machine
|
|
|
|
|
// types, a virtual CPU is implemented as a single
|
|
|
|
|
// hyperthread on a 2.6GHz Intel Sandy Bridge Xeon or
|
|
|
|
|
// Intel Ivy Bridge Xeon (or newer) processor. This
|
|
|
|
|
// means that the n1-standard-2 machine type will see
|
|
|
|
|
// a whole physical core."
|
|
|
|
|
//
|
|
|
|
|
// ... so we used n1-highcpu-2 (1.80 RAM, still
|
|
|
|
|
// plenty), just so we can get 1 whole core for the
|
|
|
|
|
// single-core Plan 9. It will see 2 virtual cores and
|
|
|
|
|
// only use 1, but we hope that 1 will be more powerful
|
|
|
|
|
// and we'll stop timing out on tests.
|
|
|
|
|
machineType: "n1-highcpu-4",
|
2016-12-02 02:15:58 +03:00
|
|
|
env: []string{"GO_TEST_TIMEOUT_SCALE=2"},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
},
|
2017-04-21 22:35:28 +03:00
|
|
|
"host-windows-amd64-2008": &HostConfig{
|
2018-07-02 20:49:55 +03:00
|
|
|
VMImage: "windows-amd64-server-2008r2-v7",
|
2017-04-21 22:35:28 +03:00
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
|
2017-08-01 02:55:28 +03:00
|
|
|
SSHUsername: "gopher",
|
2017-04-21 22:35:28 +03:00
|
|
|
},
|
|
|
|
|
"host-windows-amd64-2012": &HostConfig{
|
2018-07-02 20:49:55 +03:00
|
|
|
VMImage: "windows-amd64-server-2012r2-v7",
|
2017-04-21 22:35:28 +03:00
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
|
2017-08-01 02:55:28 +03:00
|
|
|
SSHUsername: "gopher",
|
2017-04-21 22:35:28 +03:00
|
|
|
},
|
|
|
|
|
"host-windows-amd64-2016": &HostConfig{
|
2018-07-02 20:49:55 +03:00
|
|
|
VMImage: "windows-amd64-server-2016-v7",
|
2017-04-21 22:35:28 +03:00
|
|
|
machineType: "n1-highcpu-4",
|
|
|
|
|
buildletURLTmpl: "http://storage.googleapis.com/$BUCKET/buildlet.windows-amd64",
|
|
|
|
|
goBootstrapURLTmpl: "https://storage.googleapis.com/$BUCKET/go1.4-windows-amd64.tar.gz",
|
2017-08-01 02:55:28 +03:00
|
|
|
SSHUsername: "gopher",
|
2017-04-21 22:35:28 +03:00
|
|
|
},
|
2018-07-24 22:59:34 +03:00
|
|
|
"host-windows-arm-iotcore": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
|
|
|
|
OwnerGithub: "jordanrh1",
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=C:\\Data\\Go"},
|
|
|
|
|
},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
"host-darwin-10_8": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
2018-02-21 04:25:11 +03:00
|
|
|
ExpectNum: 0,
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Notes: "MacStadium OS X 10.8 VM under VMWare ESXi",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
|
|
|
|
|
},
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
ReverseAliases: []string{"darwin-amd64-10_8"},
|
|
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
HermeticReverse: false, // TODO: make it so, like 10.12
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
},
|
|
|
|
|
"host-darwin-10_10": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
2018-02-21 04:25:11 +03:00
|
|
|
ExpectNum: 1,
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Notes: "MacStadium OS X 10.10 VM under VMWare ESXi",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
|
|
|
|
|
},
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
ReverseAliases: []string{"darwin-amd64-10_10"},
|
|
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
HermeticReverse: false, // TODO: make it so, like 10.12
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
},
|
|
|
|
|
"host-darwin-10_11": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
2018-02-21 04:25:11 +03:00
|
|
|
ExpectNum: 17,
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Notes: "MacStadium OS X 10.11 VM under VMWare ESXi",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
|
|
|
|
|
},
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
ReverseAliases: []string{"darwin-amd64-10_11"},
|
|
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
HermeticReverse: false, // TODO: make it so, like 10.12
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
},
|
2017-01-19 01:45:12 +03:00
|
|
|
"host-darwin-10_12": &HostConfig{
|
|
|
|
|
IsReverse: true,
|
2017-02-24 20:50:49 +03:00
|
|
|
ExpectNum: 2,
|
2017-01-19 01:45:12 +03:00
|
|
|
Notes: "MacStadium OS X 10.12 VM under VMWare ESXi",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/Users/gopher/go1.4",
|
|
|
|
|
},
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
ReverseAliases: []string{"darwin-amd64-10_12"},
|
|
|
|
|
SSHUsername: "gopher",
|
|
|
|
|
HermeticReverse: true, // we destroy the VM when done & let cmd/makemac recreate
|
2017-01-19 01:45:12 +03:00
|
|
|
},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
"host-linux-s390x": &HostConfig{
|
|
|
|
|
Notes: "run by IBM",
|
2017-04-13 22:38:46 +03:00
|
|
|
OwnerGithub: "mundaym",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
IsReverse: true,
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/var/buildlet/go-linux-s390x-bootstrap"},
|
|
|
|
|
ReverseAliases: []string{"linux-s390x-ibm"},
|
|
|
|
|
},
|
2016-10-06 23:30:58 +03:00
|
|
|
"host-linux-ppc64-osu": &HostConfig{
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
Notes: "Debian jessie; run by Go team on osuosl.org",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 5,
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
|
|
|
|
|
ReverseAliases: []string{"linux-ppc64-buildlet"},
|
|
|
|
|
SSHUsername: "debian",
|
|
|
|
|
HermeticReverse: false, // TODO: use rundockerbuildlet like arm64
|
2016-10-06 23:30:58 +03:00
|
|
|
},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
"host-linux-ppc64le-osu": &HostConfig{
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
Notes: "Debian jessie; run by Go team on osuosl.org",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 5,
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
|
|
|
|
|
ReverseAliases: []string{"linux-ppc64le-buildlet"},
|
|
|
|
|
SSHUsername: "debian",
|
|
|
|
|
HermeticReverse: false, // TODO: use rundockerbuildlet like arm64
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
},
|
|
|
|
|
"host-linux-arm64-linaro": &HostConfig{
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
Notes: "Ubuntu xenial; run by Go team, from linaro",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
HermeticReverse: true,
|
|
|
|
|
ExpectNum: 5,
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
|
|
|
|
|
ReverseAliases: []string{"linux-arm64-buildlet"},
|
|
|
|
|
SSHUsername: "root",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
},
|
2017-04-07 01:16:28 +03:00
|
|
|
"host-linux-arm64-packet": &HostConfig{
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
Notes: "On 96 core packet.net host (Xenial) in Docker containers (Jessie); run by Go team. See x/build/env/linux-arm64/packet",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
HermeticReverse: true,
|
|
|
|
|
ExpectNum: 20,
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"},
|
|
|
|
|
SSHUsername: "root",
|
2017-04-07 01:16:28 +03:00
|
|
|
},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
"host-solaris-amd64": &HostConfig{
|
|
|
|
|
Notes: "run by Go team on Joyent, on a SmartOS 'infrastructure container'",
|
|
|
|
|
IsReverse: true,
|
2017-02-24 20:50:49 +03:00
|
|
|
ExpectNum: 5,
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/root/go-solaris-amd64-bootstrap"},
|
|
|
|
|
ReverseAliases: []string{"solaris-amd64-smartosbuildlet"},
|
|
|
|
|
},
|
2017-07-07 23:41:20 +03:00
|
|
|
"host-solaris-oracle-amd64-oraclerel": &HostConfig{
|
|
|
|
|
Notes: "Oracle Solaris amd64 Release System",
|
|
|
|
|
Owner: "shawn.walker@oracle.com",
|
|
|
|
|
OwnerGithub: "binarycrusader",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
2017-07-31 23:58:19 +03:00
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/opt/golang/go-solaris-amd64-bootstrap"},
|
2017-07-07 23:41:20 +03:00
|
|
|
},
|
2017-06-27 20:11:15 +03:00
|
|
|
"host-solaris-oracle-shawn": &HostConfig{
|
|
|
|
|
Notes: "Oracle Solaris amd64 Development System",
|
|
|
|
|
Owner: "shawn.walker@oracle.com",
|
|
|
|
|
OwnerGithub: "binarycrusader",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
2017-07-31 23:58:19 +03:00
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/opt/golang/go-solaris-amd64-bootstrap"},
|
2017-06-27 20:11:15 +03:00
|
|
|
},
|
2016-11-03 20:35:15 +03:00
|
|
|
"host-linux-mips": &HostConfig{
|
2017-04-13 22:38:46 +03:00
|
|
|
Notes: "Run by Brendan Kirby, imgtec.com",
|
|
|
|
|
OwnerGithub: "MIPSbkirby",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
2016-11-09 21:53:36 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mips",
|
|
|
|
|
"GOARCH=mips",
|
|
|
|
|
"GOHOSTARCH=mips",
|
2016-12-10 22:10:49 +03:00
|
|
|
"GO_TEST_TIMEOUT_SCALE=4",
|
2016-11-09 21:53:36 +03:00
|
|
|
},
|
2016-11-03 20:35:15 +03:00
|
|
|
ReverseAliases: []string{"linux-mips"},
|
|
|
|
|
},
|
|
|
|
|
"host-linux-mipsle": &HostConfig{
|
2017-04-13 22:38:46 +03:00
|
|
|
Notes: "Run by Brendan Kirby, imgtec.com",
|
|
|
|
|
OwnerGithub: "MIPSbkirby",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
2016-11-09 21:53:36 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mipsle",
|
|
|
|
|
"GOARCH=mipsle",
|
|
|
|
|
"GOHOSTARCH=mipsle",
|
|
|
|
|
},
|
2016-11-03 20:35:15 +03:00
|
|
|
ReverseAliases: []string{"linux-mipsle"},
|
|
|
|
|
},
|
|
|
|
|
"host-linux-mips64": &HostConfig{
|
2017-04-13 22:38:46 +03:00
|
|
|
Notes: "Run by Brendan Kirby, imgtec.com",
|
|
|
|
|
OwnerGithub: "MIPSbkirby",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
2016-11-09 21:53:36 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mips64",
|
|
|
|
|
"GOARCH=mips64",
|
|
|
|
|
"GOHOSTARCH=mips64",
|
2016-12-10 22:10:49 +03:00
|
|
|
"GO_TEST_TIMEOUT_SCALE=4",
|
2016-11-09 21:53:36 +03:00
|
|
|
},
|
2016-11-03 20:35:15 +03:00
|
|
|
ReverseAliases: []string{"linux-mips64"},
|
|
|
|
|
},
|
|
|
|
|
"host-linux-mips64le": &HostConfig{
|
2017-04-13 22:38:46 +03:00
|
|
|
Notes: "Run by Brendan Kirby, imgtec.com",
|
|
|
|
|
OwnerGithub: "MIPSbkirby",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
2016-11-09 21:53:36 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mips64le",
|
|
|
|
|
"GOARCH=mips64le",
|
|
|
|
|
"GOHOSTARCH=mips64le",
|
|
|
|
|
},
|
2016-11-03 20:35:15 +03:00
|
|
|
ReverseAliases: []string{"linux-mips64le"},
|
|
|
|
|
},
|
2017-03-31 11:08:01 +03:00
|
|
|
"host-darwin-amd64-eliasnaur-android": &HostConfig{
|
2017-04-13 22:38:46 +03:00
|
|
|
Notes: "Mac Mini hosted by Elias Naur, running the android reverse buildlet",
|
|
|
|
|
OwnerGithub: "eliasnaur",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
2017-03-31 11:08:01 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap",
|
|
|
|
|
"GOHOSTARCH=amd64",
|
|
|
|
|
"GOOS=android",
|
|
|
|
|
},
|
|
|
|
|
},
|
2017-08-28 20:11:57 +03:00
|
|
|
"host-darwin-amd64-eliasnaur-ios": &HostConfig{
|
|
|
|
|
Notes: "Mac Mini hosted by Elias Naur, running the ios reverse buildlet",
|
2017-04-13 22:38:46 +03:00
|
|
|
OwnerGithub: "eliasnaur",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
2017-03-30 00:04:40 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/usr/local/go-bootstrap",
|
|
|
|
|
"GOHOSTARCH=amd64",
|
|
|
|
|
},
|
|
|
|
|
},
|
2018-07-25 16:14:32 +03:00
|
|
|
"host-darwin-amd64-zenly-ios": &HostConfig{
|
|
|
|
|
Notes: "MacBook Pro hosted by Zenly, running the ios reverse buildlet",
|
|
|
|
|
OwnerGithub: "znly",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOROOT_BOOTSTRAP=/usr/local/Cellar/1.10.3/libexec",
|
|
|
|
|
"GOHOSTARCH=amd64",
|
|
|
|
|
},
|
|
|
|
|
},
|
2018-08-24 21:41:05 +03:00
|
|
|
"host-aix-ppc64-osuosl": &HostConfig{
|
|
|
|
|
Notes: "AIX 7.2 VM on OSU; run by Tony Reix",
|
|
|
|
|
OwnerGithub: "trex58",
|
|
|
|
|
IsReverse: true,
|
|
|
|
|
ExpectNum: 1,
|
|
|
|
|
env: []string{"GOROOT_BOOTSTRAP=/opt/freeware/lib/golang"},
|
|
|
|
|
},
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func init() {
|
|
|
|
|
for key, c := range Hosts {
|
|
|
|
|
if key == "" {
|
|
|
|
|
panic("empty string key in Hosts")
|
|
|
|
|
}
|
|
|
|
|
if c.HostType == "" {
|
|
|
|
|
c.HostType = key
|
|
|
|
|
}
|
|
|
|
|
if c.HostType != key {
|
|
|
|
|
panic(fmt.Sprintf("HostType %q != key %q", c.HostType, key))
|
|
|
|
|
}
|
|
|
|
|
nSet := 0
|
|
|
|
|
if c.VMImage != "" {
|
|
|
|
|
nSet++
|
|
|
|
|
}
|
2018-05-05 19:36:05 +03:00
|
|
|
if c.ContainerImage != "" {
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
nSet++
|
|
|
|
|
}
|
|
|
|
|
if c.IsReverse {
|
|
|
|
|
nSet++
|
|
|
|
|
}
|
|
|
|
|
if nSet != 1 {
|
2018-05-05 19:36:05 +03:00
|
|
|
panic(fmt.Sprintf("exactly one of VMImage, ContainerImage, IsReverse must be set for host %q; got %v", key, nSet))
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
}
|
2018-05-05 19:36:05 +03:00
|
|
|
if c.buildletURLTmpl == "" && (c.VMImage != "" || c.ContainerImage != "") {
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
panic(fmt.Sprintf("missing buildletURLTmpl for host type %q", key))
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// A HostConfig describes the available ways to obtain buildlets of
|
|
|
|
|
// different types. Some host configs can server multiple
|
2018-05-14 20:22:45 +03:00
|
|
|
// builders. For example, a host config of "host-linux-jessie" can
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
// serve linux-amd64, linux-amd64-race, linux-386, linux-386-387, etc.
|
|
|
|
|
type HostConfig struct {
|
|
|
|
|
// HostType is the unique name of this host config. It is also
|
|
|
|
|
// the key in the Hosts map.
|
|
|
|
|
HostType string
|
|
|
|
|
|
|
|
|
|
// buildletURLTmpl is the URL "template" ($BUCKET is auto-expanded)
|
|
|
|
|
// for the URL to the buildlet binary.
|
2018-05-04 05:42:17 +03:00
|
|
|
// This field is required for VM and Container builders. It's not
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
// needed for reverse buildlets because in that case, the buildlets
|
|
|
|
|
// are already running and their stage0 should know how to update it
|
|
|
|
|
// it automatically.
|
|
|
|
|
buildletURLTmpl string
|
|
|
|
|
|
|
|
|
|
// Exactly 1 of these must be set:
|
2018-05-05 19:36:05 +03:00
|
|
|
VMImage string // e.g. "openbsd-amd64-60"
|
|
|
|
|
ContainerImage string // e.g. "linux-buildlet-std:latest" (suffix after "gcr.io/<PROJ>/")
|
|
|
|
|
IsReverse bool // if true, only use the reverse buildlet pool
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
|
|
|
|
|
// GCE options, if VMImage != ""
|
2015-01-15 23:46:22 +03:00
|
|
|
machineType string // optional GCE instance type
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
RegularDisk bool // if true, use spinning disk instead of SSD
|
|
|
|
|
|
2017-02-24 20:50:49 +03:00
|
|
|
// ReverseOptions:
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
ExpectNum int // expected number of reverse buildlets of this type
|
|
|
|
|
HermeticReverse bool // whether reverse buildlet has fresh env per conn
|
2017-02-24 20:50:49 +03:00
|
|
|
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
// Optional base env. GOROOT_BOOTSTRAP should go here if the buildlet
|
|
|
|
|
// has Go 1.4+ baked in somewhere.
|
|
|
|
|
env []string
|
2016-03-22 01:05:52 +03:00
|
|
|
|
|
|
|
|
// These template URLs may contain $BUCKET which is expanded to the
|
|
|
|
|
// relevant Cloud Storage bucket as specified by the build environment.
|
|
|
|
|
goBootstrapURLTmpl string // optional URL to a built Go 1.4+ tar.gz
|
2015-01-15 23:46:22 +03:00
|
|
|
|
2017-04-13 22:38:46 +03:00
|
|
|
Owner string // optional email of owner; "bradfitz@golang.org", empty means golang-dev
|
|
|
|
|
OwnerGithub string // optional GitHub username of owner
|
|
|
|
|
Notes string // notes for humans
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
SSHUsername string // username to ssh as, empty means not supported
|
|
|
|
|
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
// ReverseAliases lists alternate names for this buildlet
|
|
|
|
|
// config, for older clients doing a reverse dial into the
|
|
|
|
|
// coordinator from outside. This prevents us from updating
|
|
|
|
|
// 75+ dedicated machines/VMs atomically, switching them to
|
|
|
|
|
// the new "host-*" names.
|
|
|
|
|
// This is only applicable if IsReverse.
|
|
|
|
|
ReverseAliases []string
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// A BuildConfig describes how to run a builder.
|
|
|
|
|
type BuildConfig struct {
|
|
|
|
|
// Name is the unique name of the builder, in the form of
|
|
|
|
|
// "GOOS-GOARCH" or "GOOS-GOARCH-suffix". For example,
|
|
|
|
|
// "darwin-386", "linux-386-387", "linux-amd64-race". Some
|
|
|
|
|
// suffixes are well-known and carry special meaning, such as
|
|
|
|
|
// "-race".
|
|
|
|
|
Name string
|
|
|
|
|
|
|
|
|
|
// HostType is the required key into the Hosts map, describing
|
|
|
|
|
// the type of host this build will run on.
|
2018-05-14 20:22:45 +03:00
|
|
|
// For example, "host-linux-jessie".
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
HostType string
|
|
|
|
|
|
|
|
|
|
Notes string // notes for humans
|
|
|
|
|
|
2017-04-12 03:35:37 +03:00
|
|
|
TryBot bool // be a trybot
|
2015-05-14 04:38:20 +03:00
|
|
|
TryOnly bool // only used for trybots, and not regular builds
|
2016-05-04 20:47:12 +03:00
|
|
|
CompileOnly bool // if true, compile tests, but don't run them
|
2016-05-06 21:55:26 +03:00
|
|
|
FlakyNet bool // network tests are flaky (try anyway, but ignore some failures)
|
2015-03-21 02:14:52 +03:00
|
|
|
|
2017-04-19 23:45:24 +03:00
|
|
|
// MaxAtOnce optionally specifies a cap of how many builds of
|
|
|
|
|
// this type can run at once. Zero means unlimited. This is a
|
|
|
|
|
// temporary measure until the build scheduler
|
2017-11-30 01:43:26 +03:00
|
|
|
// (golang.org/issue/19178) is done, at which point this field
|
|
|
|
|
// should be deleted.
|
2017-04-19 23:45:24 +03:00
|
|
|
MaxAtOnce int
|
|
|
|
|
|
2017-04-13 07:34:42 +03:00
|
|
|
// SkipSnapshot, if true, means to not fetch a tarball
|
|
|
|
|
// snapshot of the world post-make.bash from the buildlet (and
|
|
|
|
|
// thus to not write it to Google Cloud Storage). This is
|
|
|
|
|
// incompatible with sharded tests, and should only be used
|
|
|
|
|
// for very slow builders or networks, unable to transfer
|
|
|
|
|
// the tarball in under ~5 minutes.
|
|
|
|
|
SkipSnapshot bool
|
|
|
|
|
|
2017-03-15 23:18:30 +03:00
|
|
|
// RunBench causes the coordinator to run benchmarks on this buildlet type.
|
|
|
|
|
RunBench bool
|
|
|
|
|
|
2016-09-23 02:29:31 +03:00
|
|
|
// StopAfterMake causes the build to stop after the make
|
|
|
|
|
// script completes, returning its result as the result of the
|
|
|
|
|
// whole build. It does not run or compile any of the tests,
|
|
|
|
|
// nor does it write a snapshot of the world to cloud
|
|
|
|
|
// storage. This option is only supported for builders whose
|
|
|
|
|
// BuildConfig.SplitMakeRun returns true.
|
|
|
|
|
StopAfterMake bool
|
|
|
|
|
|
2017-04-22 02:41:12 +03:00
|
|
|
// InstallRacePackages controls which packages to "go install
|
|
|
|
|
// -race <pkgs>" after running make.bash (or equivalent). If
|
|
|
|
|
// the builder ends in "-race", the default if non-nil is just
|
|
|
|
|
// "std".
|
|
|
|
|
InstallRacePackages []string
|
|
|
|
|
|
2017-05-20 00:23:14 +03:00
|
|
|
// GoDeps is a list of of git sha1 commits that must be in the
|
|
|
|
|
// commit to be tested's history. If absent, this builder is
|
|
|
|
|
// not run for that commit.
|
|
|
|
|
GoDeps []string
|
|
|
|
|
|
2017-12-07 03:40:40 +03:00
|
|
|
// ShouldRunDistTest optionally specifies a function which
|
|
|
|
|
// controls whether a test (a name from "go tool dist test
|
|
|
|
|
// -list") is run. The isTry value is true for trybot runs.
|
|
|
|
|
// A few general special cases are handled in
|
|
|
|
|
// cmd/coordinator's in buildStatus.shouldSkipTest.
|
|
|
|
|
ShouldRunDistTest func(distTestName string, isTry bool) bool
|
|
|
|
|
|
cmd/coordinator, dashboard: remove some trybots, shard others wider
I'm aiming to have trybot runs finish in under 5 minutes.
This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set.
openbsd-386-gce58 is the slowest builder. It has an average speed of
722 seconds (and 95 percentile of 923 seconds) over the past week, and
that's sharded over 4 machines. Too slow. It's not worth the resources
to keep it as a trybot. It hasn't caught any interesting bugs. This
builder will still run, but not as a pre-submit trybot.
freebsd-386-gce101 is not slow, but we're removing it to shift its
resources to shard other builders wider.
The coordinator now supports varying the build sharding width based on
whether a build is for a trybot or not. This CL defines separate
numbers for each, sharding builds wider as needed for some trybots.
freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down
to 3 when not in try runs.
linux-amd64-race gets one more machine during try runs, and one fewer
in regular runs.
linux-arm goes from 7 machines always, to 3 or 8, depending on whether
it's a try run.
openbsd-amd64-58 goes from 4 to 3 or 6.
windows-amd64-gce goes from 4 to 2 or 6.
windows-amd64-race goes from 4 to 2 or 6.
darwin-amd64-10_11 goes from 3 to 3 or 4.
I'll see how these do over the next few days and readjust as needed.
Also in this CL: fix the constants for the expected duration of
make.bash, which impact when we schedule the creation of test sharding
helper buildlets. We were creating them too early before, wasting
resources.
Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f
Reviewed-on: https://go-review.googlesource.com/29116
Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
|
|
|
// numTestHelpers is the number of _additional_ buildlets
|
|
|
|
|
// past the first one to help out with sharded tests.
|
|
|
|
|
// For trybots, the numTryHelpers value is used, unless it's
|
|
|
|
|
// zero, in which case numTestHelpers is used.
|
|
|
|
|
numTestHelpers int
|
|
|
|
|
numTryTestHelpers int // for trybots. if 0, numTesthelpers is used
|
2015-05-28 07:51:25 +03:00
|
|
|
|
2015-06-05 04:25:50 +03:00
|
|
|
env []string // extra environment ("key=value") pairs
|
|
|
|
|
allScriptArgs []string
|
2015-01-15 23:46:22 +03:00
|
|
|
}
|
|
|
|
|
|
2015-08-04 05:19:02 +03:00
|
|
|
func (c *BuildConfig) Env() []string {
|
2016-05-06 21:55:26 +03:00
|
|
|
env := []string{"GO_BUILDER_NAME=" + c.Name}
|
|
|
|
|
if c.FlakyNet {
|
|
|
|
|
env = append(env, "GO_BUILDER_FLAKY_NET=1")
|
|
|
|
|
}
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
env = append(env, c.hostConf().env...)
|
2016-05-06 21:55:26 +03:00
|
|
|
return append(env, c.env...)
|
2015-08-04 05:19:02 +03:00
|
|
|
}
|
2015-01-22 02:15:48 +03:00
|
|
|
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
func (c *BuildConfig) IsReverse() bool { return c.hostConf().IsReverse }
|
|
|
|
|
|
2018-05-05 19:36:05 +03:00
|
|
|
func (c *BuildConfig) IsContainer() bool { return c.hostConf().IsContainer() }
|
|
|
|
|
func (c *HostConfig) IsContainer() bool { return c.ContainerImage != "" }
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
|
2018-05-05 19:36:05 +03:00
|
|
|
func (c *BuildConfig) IsVM() bool { return c.hostConf().IsVM() }
|
|
|
|
|
func (c *HostConfig) IsVM() bool { return c.VMImage != "" }
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
|
2015-01-16 03:29:16 +03:00
|
|
|
func (c *BuildConfig) GOOS() string { return c.Name[:strings.Index(c.Name, "-")] }
|
|
|
|
|
|
|
|
|
|
func (c *BuildConfig) GOARCH() string {
|
|
|
|
|
arch := c.Name[strings.Index(c.Name, "-")+1:]
|
|
|
|
|
i := strings.Index(arch, "-")
|
|
|
|
|
if i == -1 {
|
|
|
|
|
return arch
|
|
|
|
|
}
|
|
|
|
|
return arch[:i]
|
|
|
|
|
}
|
|
|
|
|
|
2015-05-28 07:51:25 +03:00
|
|
|
// FilePathJoin is mostly like filepath.Join (without the cleaning) except
|
|
|
|
|
// it uses the path separator of c.GOOS instead of the host system's.
|
|
|
|
|
func (c *BuildConfig) FilePathJoin(x ...string) string {
|
|
|
|
|
if c.GOOS() == "windows" {
|
|
|
|
|
return strings.Join(x, "\\")
|
|
|
|
|
}
|
|
|
|
|
return strings.Join(x, "/")
|
|
|
|
|
}
|
|
|
|
|
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
func (c *BuildConfig) hostConf() *HostConfig {
|
|
|
|
|
if c, ok := Hosts[c.HostType]; ok {
|
|
|
|
|
return c
|
|
|
|
|
}
|
|
|
|
|
panic(fmt.Sprintf("missing buildlet config for buildlet %q", c.Name))
|
|
|
|
|
}
|
|
|
|
|
|
2016-03-22 01:05:52 +03:00
|
|
|
// BuildletBinaryURL returns the public URL of this builder's buildlet.
|
|
|
|
|
func (c *BuildConfig) GoBootstrapURL(e *buildenv.Environment) string {
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
return strings.Replace(c.hostConf().goBootstrapURLTmpl, "$BUCKET", e.BuildletBucket, 1)
|
2016-03-22 01:05:52 +03:00
|
|
|
}
|
|
|
|
|
|
2015-01-28 01:22:21 +03:00
|
|
|
// BuildletBinaryURL returns the public URL of this builder's buildlet.
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
func (c *HostConfig) BuildletBinaryURL(e *buildenv.Environment) string {
|
2016-03-22 01:05:52 +03:00
|
|
|
tmpl := c.buildletURLTmpl
|
2016-02-15 02:59:59 +03:00
|
|
|
return strings.Replace(tmpl, "$BUCKET", e.BuildletBucket, 1)
|
2015-01-28 01:22:21 +03:00
|
|
|
}
|
|
|
|
|
|
2015-06-17 18:22:14 +03:00
|
|
|
func (c *BuildConfig) IsRace() bool {
|
|
|
|
|
return strings.HasSuffix(c.Name, "-race")
|
|
|
|
|
}
|
|
|
|
|
|
2017-04-22 02:41:12 +03:00
|
|
|
func (c *BuildConfig) GoInstallRacePackages() []string {
|
|
|
|
|
if c.InstallRacePackages != nil {
|
|
|
|
|
return append([]string(nil), c.InstallRacePackages...)
|
|
|
|
|
}
|
|
|
|
|
if c.IsRace() {
|
|
|
|
|
return []string{"std"}
|
|
|
|
|
}
|
|
|
|
|
return nil
|
|
|
|
|
}
|
|
|
|
|
|
2015-01-16 20:54:03 +03:00
|
|
|
// AllScript returns the relative path to the operating system's script to
|
|
|
|
|
// do the build and run its standard set of tests.
|
|
|
|
|
// Example values are "src/all.bash", "src/all.bat", "src/all.rc".
|
|
|
|
|
func (c *BuildConfig) AllScript() string {
|
2017-04-13 00:37:13 +03:00
|
|
|
if c.Name == "" {
|
|
|
|
|
panic("bogus BuildConfig")
|
|
|
|
|
}
|
2015-06-17 18:22:14 +03:00
|
|
|
if c.IsRace() {
|
2015-01-22 02:15:48 +03:00
|
|
|
if strings.HasPrefix(c.Name, "windows-") {
|
|
|
|
|
return "src/race.bat"
|
|
|
|
|
}
|
|
|
|
|
return "src/race.bash"
|
|
|
|
|
}
|
2015-01-16 20:54:03 +03:00
|
|
|
if strings.HasPrefix(c.Name, "windows-") {
|
|
|
|
|
return "src/all.bat"
|
|
|
|
|
}
|
|
|
|
|
if strings.HasPrefix(c.Name, "plan9-") {
|
|
|
|
|
return "src/all.rc"
|
|
|
|
|
}
|
2015-02-19 01:12:22 +03:00
|
|
|
if strings.HasPrefix(c.Name, "nacl-") {
|
|
|
|
|
return "src/nacltest.bash"
|
|
|
|
|
}
|
2015-09-03 20:35:10 +03:00
|
|
|
if strings.HasPrefix(c.Name, "android-") {
|
|
|
|
|
return "src/androidtest.bash"
|
|
|
|
|
}
|
2015-04-29 15:54:19 +03:00
|
|
|
if strings.HasPrefix(c.Name, "darwin-arm") {
|
|
|
|
|
return "src/iostest.bash"
|
|
|
|
|
}
|
2016-05-05 03:42:49 +03:00
|
|
|
if strings.HasPrefix(c.Name, "misc-compile") {
|
2015-04-30 00:44:46 +03:00
|
|
|
return "src/buildall.bash"
|
|
|
|
|
}
|
2015-01-16 20:54:03 +03:00
|
|
|
return "src/all.bash"
|
|
|
|
|
}
|
|
|
|
|
|
2015-05-27 21:51:27 +03:00
|
|
|
// SplitMakeRun reports whether the coordinator should first compile
|
|
|
|
|
// (using c.MakeScript), then snapshot, then run the tests (ideally
|
|
|
|
|
// sharded) using c.RunScript.
|
|
|
|
|
// Eventually this function should always return true (and then be deleted)
|
|
|
|
|
// but for now we've only set up the scripts and verified that the main
|
|
|
|
|
// configurations work.
|
|
|
|
|
func (c *BuildConfig) SplitMakeRun() bool {
|
|
|
|
|
switch c.AllScript() {
|
2016-05-04 20:47:12 +03:00
|
|
|
case "src/all.bash", "src/all.bat",
|
|
|
|
|
"src/race.bash", "src/race.bat",
|
|
|
|
|
"src/all.rc",
|
|
|
|
|
"src/nacltest.bash":
|
2015-05-27 21:51:27 +03:00
|
|
|
// These we've verified to work.
|
|
|
|
|
return true
|
|
|
|
|
}
|
2016-05-04 20:47:12 +03:00
|
|
|
// TODO(bradfitz): make androidtest.bash and iotest.bash work
|
|
|
|
|
// too. And buildall.bash should really just be N small
|
2018-05-04 05:42:17 +03:00
|
|
|
// container jobs instead of a "buildall.bash". Then we can
|
2016-05-04 20:47:12 +03:00
|
|
|
// delete this whole method.
|
2015-05-27 21:51:27 +03:00
|
|
|
return false
|
|
|
|
|
}
|
|
|
|
|
|
2015-06-05 02:45:17 +03:00
|
|
|
func (c *BuildConfig) BuildSubrepos() bool {
|
|
|
|
|
if !c.SplitMakeRun() {
|
|
|
|
|
return false
|
|
|
|
|
}
|
2018-09-28 02:43:25 +03:00
|
|
|
// TODO(bradfitz,dmitshur): move this into BuildConfig bools, rather than this Name switch.
|
2015-06-11 20:01:24 +03:00
|
|
|
switch c.Name {
|
2017-12-07 03:40:40 +03:00
|
|
|
case "darwin-amd64-10_11",
|
2016-09-07 08:41:02 +03:00
|
|
|
"darwin-386-10_11",
|
2017-12-07 03:40:40 +03:00
|
|
|
// TODO: add darwin-amd64-10_12 when we have a build scheduler
|
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1.
freebsd.org says that the current supported releases are:
> Production: 11.1, 11.0, 10.4, 10.3
Previously we had builders for 9.3, 10.1, and 11.0.
We'll keep FreeBSD 9.3 for now (as it's still supported by the last
two released Go versions), but we'll stop testing FreeBSD 9.3 at tip
shortly here (that will be a separate change).
This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3,
spanning the range from the old to newest supported release. We'll
stop testing 10.4 and 11.0, as that's a bit overkill.
Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9
release notes (https://golang.org/doc/go1.9#freebsd) said that future
Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test
FreeBSD 10.3, being the oldest. (Go has historically "supported"
FreeBSD releases a bit longer than they're officially supported by
FreeBSD, partly due to our out-of-sync release cycles.)
Fixes golang/go#19303
Fixes golang/go#22854
Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f
Reviewed-on: https://go-review.googlesource.com/80297
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
|
|
|
"freebsd-amd64-93",
|
|
|
|
|
"freebsd-386-10_3", "freebsd-amd64-10_3",
|
|
|
|
|
"freebsd-386-11_1", "freebsd-amd64-11_1",
|
2018-09-28 02:43:25 +03:00
|
|
|
"freebsd-386-10_4", "freebsd-amd64-10_4",
|
|
|
|
|
"freebsd-386-11_2", "freebsd-amd64-11_2",
|
2015-06-11 20:01:24 +03:00
|
|
|
"linux-386", "linux-amd64", "linux-amd64-nocgo",
|
2018-08-01 23:08:56 +03:00
|
|
|
"linux-s390x-ibm",
|
2016-12-15 01:41:25 +03:00
|
|
|
"openbsd-386-60", "openbsd-amd64-60",
|
2017-11-24 21:14:33 +03:00
|
|
|
"openbsd-386-62", "openbsd-amd64-62",
|
2018-04-28 01:47:56 +03:00
|
|
|
"netbsd-amd64-8_0",
|
|
|
|
|
"netbsd-386-8_0",
|
2015-06-11 20:01:24 +03:00
|
|
|
"plan9-386",
|
2017-08-09 03:53:04 +03:00
|
|
|
"freebsd-arm-paulzhol",
|
2017-07-27 07:52:12 +03:00
|
|
|
"windows-amd64-2016", "windows-386-2008":
|
2015-06-11 20:01:24 +03:00
|
|
|
return true
|
|
|
|
|
default:
|
|
|
|
|
return false
|
|
|
|
|
}
|
2015-06-05 02:45:17 +03:00
|
|
|
}
|
|
|
|
|
|
2015-05-21 07:02:38 +03:00
|
|
|
// AllScriptArgs returns the set of arguments that should be passed to the
|
2015-04-29 15:54:19 +03:00
|
|
|
// all.bash-equivalent script. Usually empty.
|
|
|
|
|
func (c *BuildConfig) AllScriptArgs() []string {
|
|
|
|
|
if strings.HasPrefix(c.Name, "darwin-arm") {
|
|
|
|
|
return []string{"-restart"}
|
|
|
|
|
}
|
2015-06-05 04:25:50 +03:00
|
|
|
return append([]string(nil), c.allScriptArgs...)
|
2015-04-29 15:54:19 +03:00
|
|
|
}
|
|
|
|
|
|
2015-02-02 15:05:01 +03:00
|
|
|
// MakeScript returns the relative path to the operating system's script to
|
|
|
|
|
// do the build.
|
|
|
|
|
// Example values are "src/make.bash", "src/make.bat", "src/make.rc".
|
|
|
|
|
func (c *BuildConfig) MakeScript() string {
|
|
|
|
|
if strings.HasPrefix(c.Name, "windows-") {
|
|
|
|
|
return "src/make.bat"
|
|
|
|
|
}
|
|
|
|
|
if strings.HasPrefix(c.Name, "plan9-") {
|
|
|
|
|
return "src/make.rc"
|
|
|
|
|
}
|
2016-05-04 20:47:12 +03:00
|
|
|
if strings.HasPrefix(c.Name, "nacl-") {
|
|
|
|
|
return "src/naclmake.bash"
|
|
|
|
|
}
|
2015-02-02 15:05:01 +03:00
|
|
|
return "src/make.bash"
|
|
|
|
|
}
|
|
|
|
|
|
2015-05-21 07:02:38 +03:00
|
|
|
// MakeScriptArgs returns the set of arguments that should be passed to the
|
|
|
|
|
// make.bash-equivalent script. Usually empty.
|
|
|
|
|
func (c *BuildConfig) MakeScriptArgs() []string {
|
|
|
|
|
return c.AllScriptArgs()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// RunScript returns the relative path to the operating system's script to
|
|
|
|
|
// run the test suite.
|
|
|
|
|
// Example values are "src/run.bash", "src/run.bat", "src/run.rc".
|
|
|
|
|
func (c *BuildConfig) RunScript() string {
|
|
|
|
|
if strings.HasPrefix(c.Name, "windows-") {
|
|
|
|
|
return "src/run.bat"
|
|
|
|
|
}
|
|
|
|
|
if strings.HasPrefix(c.Name, "plan9-") {
|
|
|
|
|
return "src/run.rc"
|
|
|
|
|
}
|
|
|
|
|
return "src/run.bash"
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// RunScriptArgs returns the set of arguments that should be passed to the
|
|
|
|
|
// run.bash-equivalent script.
|
|
|
|
|
func (c *BuildConfig) RunScriptArgs() []string {
|
|
|
|
|
return []string{"--no-rebuild"}
|
|
|
|
|
}
|
|
|
|
|
|
2015-02-02 15:05:01 +03:00
|
|
|
// GorootFinal returns the default install location for
|
|
|
|
|
// releases for this platform.
|
|
|
|
|
func (c *BuildConfig) GorootFinal() string {
|
|
|
|
|
if strings.HasPrefix(c.Name, "windows-") {
|
|
|
|
|
return "c:\\go"
|
|
|
|
|
}
|
|
|
|
|
return "/usr/local/go"
|
|
|
|
|
}
|
|
|
|
|
|
2015-01-15 23:46:22 +03:00
|
|
|
// MachineType returns the GCE machine type to use for this builder.
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
func (c *HostConfig) MachineType() string {
|
2015-01-15 23:46:22 +03:00
|
|
|
if v := c.machineType; v != "" {
|
|
|
|
|
return v
|
|
|
|
|
}
|
2018-05-04 05:42:17 +03:00
|
|
|
if c.IsContainer() {
|
|
|
|
|
// Set a higher default machine size for containers,
|
|
|
|
|
// so their /workdir tmpfs can be larger. The COS
|
|
|
|
|
// image has no swap, so we want to make sure the
|
|
|
|
|
// /workdir fits completely in memory.
|
|
|
|
|
return "n1-standard-4" // 4 CPUs, 15GB RAM
|
|
|
|
|
}
|
2015-01-15 23:46:22 +03:00
|
|
|
return "n1-highcpu-2"
|
|
|
|
|
}
|
|
|
|
|
|
2015-05-01 03:03:01 +03:00
|
|
|
// ShortOwner returns a short human-readable owner.
|
|
|
|
|
func (c BuildConfig) ShortOwner() string {
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
owner := c.hostConf().Owner
|
|
|
|
|
if owner == "" {
|
2015-05-01 03:03:01 +03:00
|
|
|
return "go-dev"
|
|
|
|
|
}
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
return strings.TrimSuffix(owner, "@golang.org")
|
2015-05-01 03:03:01 +03:00
|
|
|
}
|
|
|
|
|
|
2017-04-13 22:38:46 +03:00
|
|
|
// OwnerGithub returns the Github handle of the owner.
|
|
|
|
|
func (c BuildConfig) OwnerGithub() string {
|
|
|
|
|
return c.hostConf().OwnerGithub
|
|
|
|
|
}
|
|
|
|
|
|
2016-09-23 03:06:28 +03:00
|
|
|
// PoolName returns a short summary of the builder's host type for the
|
2017-04-03 17:51:28 +03:00
|
|
|
// https://farmer.golang.org/builders page.
|
2016-09-23 03:06:28 +03:00
|
|
|
func (c *HostConfig) PoolName() string {
|
|
|
|
|
switch {
|
|
|
|
|
case c.IsReverse:
|
|
|
|
|
return "Reverse (dedicated machine/VM)"
|
2018-05-05 19:36:05 +03:00
|
|
|
case c.IsVM():
|
2016-09-23 03:06:28 +03:00
|
|
|
return "GCE VM"
|
2018-05-05 19:36:05 +03:00
|
|
|
case c.IsContainer():
|
2018-05-04 05:42:17 +03:00
|
|
|
return "Container"
|
2016-09-23 03:06:28 +03:00
|
|
|
}
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
panic("unknown builder type")
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// IsHermetic reports whether this host config gets a fresh
|
|
|
|
|
// environment (including /usr, /var, etc) for each execution. This is
|
|
|
|
|
// true for VMs, GKE, and reverse buildlets running their containers
|
|
|
|
|
// running in Docker, but false on some reverse buildlets.
|
|
|
|
|
func (c *HostConfig) IsHermetic() bool {
|
|
|
|
|
switch {
|
|
|
|
|
case c.IsReverse:
|
|
|
|
|
return c.HermeticReverse
|
2018-05-05 19:36:05 +03:00
|
|
|
case c.IsVM():
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
return true
|
2018-05-05 19:36:05 +03:00
|
|
|
case c.IsContainer():
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
return true
|
|
|
|
|
}
|
|
|
|
|
panic("unknown builder type")
|
2016-09-23 03:06:28 +03:00
|
|
|
}
|
|
|
|
|
|
2015-06-05 04:25:50 +03:00
|
|
|
// GCENumCPU reports the number of GCE CPUs this buildlet requires.
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
func (c *HostConfig) GCENumCPU() int {
|
2015-06-05 04:25:50 +03:00
|
|
|
t := c.MachineType()
|
|
|
|
|
n, _ := strconv.Atoi(t[strings.LastIndex(t, "-")+1:])
|
|
|
|
|
return n
|
|
|
|
|
}
|
|
|
|
|
|
cmd/coordinator, dashboard: remove some trybots, shard others wider
I'm aiming to have trybot runs finish in under 5 minutes.
This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set.
openbsd-386-gce58 is the slowest builder. It has an average speed of
722 seconds (and 95 percentile of 923 seconds) over the past week, and
that's sharded over 4 machines. Too slow. It's not worth the resources
to keep it as a trybot. It hasn't caught any interesting bugs. This
builder will still run, but not as a pre-submit trybot.
freebsd-386-gce101 is not slow, but we're removing it to shift its
resources to shard other builders wider.
The coordinator now supports varying the build sharding width based on
whether a build is for a trybot or not. This CL defines separate
numbers for each, sharding builds wider as needed for some trybots.
freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down
to 3 when not in try runs.
linux-amd64-race gets one more machine during try runs, and one fewer
in regular runs.
linux-arm goes from 7 machines always, to 3 or 8, depending on whether
it's a try run.
openbsd-amd64-58 goes from 4 to 3 or 6.
windows-amd64-gce goes from 4 to 2 or 6.
windows-amd64-race goes from 4 to 2 or 6.
darwin-amd64-10_11 goes from 3 to 3 or 4.
I'll see how these do over the next few days and readjust as needed.
Also in this CL: fix the constants for the expected duration of
make.bash, which impact when we schedule the creation of test sharding
helper buildlets. We were creating them too early before, wasting
resources.
Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f
Reviewed-on: https://go-review.googlesource.com/29116
Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
|
|
|
func (c *BuildConfig) NumTestHelpers(isTry bool) int {
|
|
|
|
|
if isTry && c.numTryTestHelpers != 0 {
|
|
|
|
|
return c.numTryTestHelpers
|
|
|
|
|
}
|
|
|
|
|
return c.numTestHelpers
|
|
|
|
|
}
|
|
|
|
|
|
2015-01-15 23:46:22 +03:00
|
|
|
func init() {
|
2015-01-22 08:16:54 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-08-04 07:25:07 +03:00
|
|
|
Name: "freebsd-amd64-gce93",
|
|
|
|
|
HostType: "host-freebsd-93-gce",
|
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1.
freebsd.org says that the current supported releases are:
> Production: 11.1, 11.0, 10.4, 10.3
Previously we had builders for 9.3, 10.1, and 11.0.
We'll keep FreeBSD 9.3 for now (as it's still supported by the last
two released Go versions), but we'll stop testing FreeBSD 9.3 at tip
shortly here (that will be a separate change).
This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3,
spanning the range from the old to newest supported release. We'll
stop testing 10.4 and 11.0, as that's a bit overkill.
Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9
release notes (https://golang.org/doc/go1.9#freebsd) said that future
Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test
FreeBSD 10.3, being the oldest. (Go has historically "supported"
FreeBSD releases a bit longer than they're officially supported by
FreeBSD, partly due to our out-of-sync release cycles.)
Fixes golang/go#19303
Fixes golang/go#22854
Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f
Reviewed-on: https://go-review.googlesource.com/80297
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
|
|
|
TryOnly: true, // don't run regular build...
|
|
|
|
|
TryBot: false, // .. and don't be a trybot. Only for gomote.
|
2017-11-30 01:43:26 +03:00
|
|
|
MaxAtOnce: 2,
|
2015-01-22 08:16:54 +03:00
|
|
|
})
|
2015-01-22 02:15:48 +03:00
|
|
|
addBuilder(BuildConfig{
|
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1.
freebsd.org says that the current supported releases are:
> Production: 11.1, 11.0, 10.4, 10.3
Previously we had builders for 9.3, 10.1, and 11.0.
We'll keep FreeBSD 9.3 for now (as it's still supported by the last
two released Go versions), but we'll stop testing FreeBSD 9.3 at tip
shortly here (that will be a separate change).
This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3,
spanning the range from the old to newest supported release. We'll
stop testing 10.4 and 11.0, as that's a bit overkill.
Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9
release notes (https://golang.org/doc/go1.9#freebsd) said that future
Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test
FreeBSD 10.3, being the oldest. (Go has historically "supported"
FreeBSD releases a bit longer than they're officially supported by
FreeBSD, partly due to our out-of-sync release cycles.)
Fixes golang/go#19303
Fixes golang/go#22854
Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f
Reviewed-on: https://go-review.googlesource.com/80297
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
|
|
|
Name: "freebsd-amd64-10_3",
|
|
|
|
|
HostType: "host-freebsd-10_3",
|
2017-11-30 01:43:26 +03:00
|
|
|
MaxAtOnce: 2,
|
2015-01-22 02:15:48 +03:00
|
|
|
})
|
2018-09-28 02:43:25 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "freebsd-amd64-10_4",
|
|
|
|
|
HostType: "host-freebsd-10_4",
|
|
|
|
|
MaxAtOnce: 2,
|
|
|
|
|
})
|
2017-02-27 19:05:25 +03:00
|
|
|
addBuilder(BuildConfig{
|
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1.
freebsd.org says that the current supported releases are:
> Production: 11.1, 11.0, 10.4, 10.3
Previously we had builders for 9.3, 10.1, and 11.0.
We'll keep FreeBSD 9.3 for now (as it's still supported by the last
two released Go versions), but we'll stop testing FreeBSD 9.3 at tip
shortly here (that will be a separate change).
This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3,
spanning the range from the old to newest supported release. We'll
stop testing 10.4 and 11.0, as that's a bit overkill.
Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9
release notes (https://golang.org/doc/go1.9#freebsd) said that future
Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test
FreeBSD 10.3, being the oldest. (Go has historically "supported"
FreeBSD releases a bit longer than they're officially supported by
FreeBSD, partly due to our out-of-sync release cycles.)
Fixes golang/go#19303
Fixes golang/go#22854
Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f
Reviewed-on: https://go-review.googlesource.com/80297
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
|
|
|
Name: "freebsd-amd64-11_1",
|
|
|
|
|
HostType: "host-freebsd-11_1",
|
2017-07-27 07:52:12 +03:00
|
|
|
TryBot: true,
|
2017-12-07 03:40:40 +03:00
|
|
|
ShouldRunDistTest: fasterTrybots,
|
2017-07-27 07:52:12 +03:00
|
|
|
numTryTestHelpers: 4,
|
2017-11-30 01:43:26 +03:00
|
|
|
MaxAtOnce: 2,
|
2017-02-27 19:05:25 +03:00
|
|
|
})
|
2018-09-28 02:43:25 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "freebsd-amd64-11_2",
|
|
|
|
|
HostType: "host-freebsd-11_2",
|
|
|
|
|
TryBot: false, // not yet. once we see it's passing regularly.
|
|
|
|
|
ShouldRunDistTest: fasterTrybots,
|
|
|
|
|
numTryTestHelpers: 4,
|
|
|
|
|
MaxAtOnce: 2,
|
|
|
|
|
})
|
2018-10-05 23:57:29 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "freebsd-amd64-12_0",
|
|
|
|
|
HostType: "host-freebsd-12_0",
|
|
|
|
|
TryBot: false, // not yet. once we see it's passing regularly.
|
|
|
|
|
ShouldRunDistTest: fasterTrybots,
|
|
|
|
|
numTryTestHelpers: 4,
|
|
|
|
|
MaxAtOnce: 2,
|
|
|
|
|
})
|
2015-01-22 02:15:48 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-08-04 07:25:07 +03:00
|
|
|
Name: "freebsd-amd64-race",
|
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1.
freebsd.org says that the current supported releases are:
> Production: 11.1, 11.0, 10.4, 10.3
Previously we had builders for 9.3, 10.1, and 11.0.
We'll keep FreeBSD 9.3 for now (as it's still supported by the last
two released Go versions), but we'll stop testing FreeBSD 9.3 at tip
shortly here (that will be a separate change).
This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3,
spanning the range from the old to newest supported release. We'll
stop testing 10.4 and 11.0, as that's a bit overkill.
Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9
release notes (https://golang.org/doc/go1.9#freebsd) said that future
Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test
FreeBSD 10.3, being the oldest. (Go has historically "supported"
FreeBSD releases a bit longer than they're officially supported by
FreeBSD, partly due to our out-of-sync release cycles.)
Fixes golang/go#19303
Fixes golang/go#22854
Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f
Reviewed-on: https://go-review.googlesource.com/80297
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
|
|
|
HostType: "host-freebsd-11_1",
|
2017-11-30 01:43:26 +03:00
|
|
|
MaxAtOnce: 2,
|
2015-01-22 02:15:48 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
dashboard, env/freebsd-amd64: update to FreeBSD 10.3 and 11.1.
freebsd.org says that the current supported releases are:
> Production: 11.1, 11.0, 10.4, 10.3
Previously we had builders for 9.3, 10.1, and 11.0.
We'll keep FreeBSD 9.3 for now (as it's still supported by the last
two released Go versions), but we'll stop testing FreeBSD 9.3 at tip
shortly here (that will be a separate change).
This CL updates our builders from 11.0 to 11.1 and 10.1 to 10.3,
spanning the range from the old to newest supported release. We'll
stop testing 10.4 and 11.0, as that's a bit overkill.
Another reason we're doing 10.3 instead of 10.4 is because the Go 1.9
release notes (https://golang.org/doc/go1.9#freebsd) said that future
Go releases (Go 1.10) will require FreeBSD 10.3+, so we need to test
FreeBSD 10.3, being the oldest. (Go has historically "supported"
FreeBSD releases a bit longer than they're officially supported by
FreeBSD, partly due to our out-of-sync release cycles.)
Fixes golang/go#19303
Fixes golang/go#22854
Change-Id: I69589000b1c87aecf054857f514cc48ccedf752f
Reviewed-on: https://go-review.googlesource.com/80297
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-11-28 20:51:41 +03:00
|
|
|
Name: "freebsd-386-10_3",
|
|
|
|
|
HostType: "host-freebsd-10_3",
|
2017-08-04 07:25:07 +03:00
|
|
|
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
|
2017-11-30 01:43:26 +03:00
|
|
|
MaxAtOnce: 2,
|
2015-01-28 01:22:21 +03:00
|
|
|
})
|
2018-09-28 02:43:25 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "freebsd-386-10_4",
|
|
|
|
|
HostType: "host-freebsd-10_4",
|
|
|
|
|
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
|
|
|
|
|
MaxAtOnce: 2,
|
|
|
|
|
})
|
2017-02-27 19:05:25 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-12-07 03:40:40 +03:00
|
|
|
Name: "freebsd-386-11_1",
|
|
|
|
|
HostType: "host-freebsd-11_1",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
|
|
|
|
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
|
|
|
|
|
MaxAtOnce: 2,
|
2017-02-27 19:05:25 +03:00
|
|
|
})
|
2018-09-28 02:43:25 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "freebsd-386-11_2",
|
|
|
|
|
HostType: "host-freebsd-11_2",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
|
|
|
|
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
|
|
|
|
|
MaxAtOnce: 2,
|
|
|
|
|
})
|
2015-01-28 01:22:21 +03:00
|
|
|
addBuilder(BuildConfig{
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Name: "linux-386",
|
2018-05-14 20:22:45 +03:00
|
|
|
HostType: "host-linux-jessie",
|
2017-12-07 03:40:40 +03:00
|
|
|
ShouldRunDistTest: fasterTrybots,
|
2017-04-12 03:35:37 +03:00
|
|
|
TryBot: true,
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
|
|
|
|
|
numTestHelpers: 1,
|
|
|
|
|
numTryTestHelpers: 3,
|
2015-01-28 01:22:21 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Name: "linux-386-387",
|
|
|
|
|
Notes: "GO386=387",
|
2018-05-14 20:22:45 +03:00
|
|
|
HostType: "host-linux-jessie",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
env: []string{"GOARCH=386", "GOHOSTARCH=386", "GO386=387"},
|
2015-01-28 01:22:21 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Name: "linux-amd64",
|
2018-05-14 20:22:45 +03:00
|
|
|
HostType: "host-linux-jessie",
|
2018-05-04 05:42:17 +03:00
|
|
|
TryBot: true,
|
|
|
|
|
MaxAtOnce: 3,
|
|
|
|
|
numTestHelpers: 1,
|
|
|
|
|
numTryTestHelpers: 4,
|
|
|
|
|
RunBench: true,
|
2015-11-20 21:00:06 +03:00
|
|
|
})
|
2017-11-18 06:31:35 +03:00
|
|
|
|
|
|
|
|
const testAlpine = false // Issue 22689 (hide all red builders), Issue 19938 (get Alpine passing)
|
|
|
|
|
if testAlpine {
|
|
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "linux-amd64-alpine",
|
|
|
|
|
HostType: "host-linux-x86-alpine",
|
|
|
|
|
})
|
|
|
|
|
}
|
|
|
|
|
|
2017-01-09 23:02:06 +03:00
|
|
|
// Add the -vetall builder. The builder name suffix "-vetall" is recognized by cmd/dist/test.go
|
|
|
|
|
// to only run the "go vet std cmd" test and no others.
|
|
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "misc-vet-vetall",
|
2018-05-14 20:22:45 +03:00
|
|
|
HostType: "host-linux-jessie",
|
2017-01-09 23:02:06 +03:00
|
|
|
Notes: "Runs vet over the standard library.",
|
2017-04-12 03:35:37 +03:00
|
|
|
TryBot: true,
|
2017-01-09 23:02:06 +03:00
|
|
|
numTestHelpers: 5,
|
|
|
|
|
})
|
2016-05-05 03:42:49 +03:00
|
|
|
|
|
|
|
|
addMiscCompile := func(suffix, rx string) {
|
|
|
|
|
addBuilder(BuildConfig{
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Name: "misc-compile" + suffix,
|
2018-05-14 20:22:45 +03:00
|
|
|
HostType: "host-linux-jessie",
|
2017-04-12 03:35:37 +03:00
|
|
|
TryBot: true,
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
TryOnly: true,
|
|
|
|
|
CompileOnly: true,
|
|
|
|
|
Notes: "Runs buildall.sh to cross-compile std packages for " + rx + ", but doesn't run any tests.",
|
2016-05-05 03:42:49 +03:00
|
|
|
allScriptArgs: []string{
|
|
|
|
|
// Filtering pattern to buildall.bash:
|
|
|
|
|
rx,
|
|
|
|
|
},
|
|
|
|
|
})
|
|
|
|
|
}
|
2018-05-24 01:19:10 +03:00
|
|
|
addMiscCompile("", "^(linux-arm64|linux-s390x|solaris-amd64|darwin-386)$") // 4 ports
|
|
|
|
|
addMiscCompile("-nacl", "^nacl") // 3
|
|
|
|
|
addMiscCompile("-mips", "^linux-mips") // 4
|
|
|
|
|
addMiscCompile("-ppc", "^linux-ppc64") // 2
|
|
|
|
|
addMiscCompile("-plan9", "^plan9-") // 3
|
|
|
|
|
addMiscCompile("-freebsd", "^freebsd-") // 3
|
|
|
|
|
addMiscCompile("-netbsd", "^netbsd-") // 3
|
|
|
|
|
addMiscCompile("-openbsd", "^openbsd-") // 3
|
2018-06-19 23:50:06 +03:00
|
|
|
// TODO: Issue 25963, get the misc-compile trybots for mobile working, and then:
|
|
|
|
|
// addMiscCompile("-mobile", "(^android|darwin-arm64)") // 5 ports
|
2016-05-05 03:42:49 +03:00
|
|
|
|
2015-01-28 01:22:21 +03:00
|
|
|
addBuilder(BuildConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Name: "linux-amd64-nocgo",
|
2018-05-14 20:22:45 +03:00
|
|
|
HostType: "host-linux-jessie",
|
2018-05-04 05:42:17 +03:00
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
Notes: "cgo disabled",
|
2015-01-28 01:22:21 +03:00
|
|
|
env: []string{
|
|
|
|
|
"CGO_ENABLED=0",
|
|
|
|
|
// This USER=root was required for Docker-based builds but probably isn't required
|
|
|
|
|
// in the VM anymore, since the buildlet probably already has this in its environment.
|
|
|
|
|
// (It was required because without cgo, it couldn't find the username)
|
|
|
|
|
"USER=root",
|
|
|
|
|
},
|
|
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Name: "linux-amd64-noopt",
|
|
|
|
|
Notes: "optimizations and inlining disabled",
|
2018-05-14 20:22:45 +03:00
|
|
|
HostType: "host-linux-jessie",
|
2018-05-04 05:42:17 +03:00
|
|
|
env: []string{"GO_GCFLAGS=-N -l"},
|
|
|
|
|
MaxAtOnce: 1,
|
2015-01-28 01:22:21 +03:00
|
|
|
})
|
2016-03-19 01:30:38 +03:00
|
|
|
addBuilder(BuildConfig{
|
2016-05-04 20:47:12 +03:00
|
|
|
Name: "linux-amd64-ssacheck",
|
2018-05-14 20:22:45 +03:00
|
|
|
HostType: "host-linux-jessie",
|
2018-05-04 05:42:17 +03:00
|
|
|
MaxAtOnce: 1,
|
2017-12-07 03:40:40 +03:00
|
|
|
TryBot: false, // TODO: add a func to conditionally run this trybot if compiler dirs are touched
|
2016-05-04 20:47:12 +03:00
|
|
|
CompileOnly: true,
|
|
|
|
|
Notes: "SSA internal checks enabled",
|
2017-03-17 07:55:29 +03:00
|
|
|
env: []string{"GO_GCFLAGS=-d=ssa/check/on,dclstack"},
|
2017-05-20 00:23:14 +03:00
|
|
|
GoDeps: []string{
|
|
|
|
|
"f65abf6ddc8d1f3d403a9195fd74eaffa022b07f", // adds dclstack
|
|
|
|
|
},
|
2016-03-19 01:30:38 +03:00
|
|
|
})
|
2017-04-22 02:41:12 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "linux-amd64-racecompile",
|
2018-05-14 20:22:45 +03:00
|
|
|
HostType: "host-linux-jessie",
|
2017-12-07 03:40:40 +03:00
|
|
|
TryBot: false, // TODO: add a func to conditionally run this trybot if compiler dirs are touched
|
2018-05-04 05:42:17 +03:00
|
|
|
MaxAtOnce: 1,
|
2017-04-22 02:41:12 +03:00
|
|
|
CompileOnly: true,
|
2017-04-25 00:14:05 +03:00
|
|
|
SkipSnapshot: true,
|
|
|
|
|
StopAfterMake: true,
|
2017-04-22 02:41:12 +03:00
|
|
|
InstallRacePackages: []string{"cmd/compile"},
|
|
|
|
|
Notes: "race-enabled cmd/compile",
|
2017-05-20 00:23:14 +03:00
|
|
|
GoDeps: []string{
|
|
|
|
|
"22f1b56dab29d397d2bdbdd603d85e60fb678089", // adds cmd/compile -c; Issue 20222
|
|
|
|
|
},
|
2017-04-22 02:41:12 +03:00
|
|
|
})
|
2015-01-28 01:22:21 +03:00
|
|
|
addBuilder(BuildConfig{
|
cmd/coordinator, dashboard: remove some trybots, shard others wider
I'm aiming to have trybot runs finish in under 5 minutes.
This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set.
openbsd-386-gce58 is the slowest builder. It has an average speed of
722 seconds (and 95 percentile of 923 seconds) over the past week, and
that's sharded over 4 machines. Too slow. It's not worth the resources
to keep it as a trybot. It hasn't caught any interesting bugs. This
builder will still run, but not as a pre-submit trybot.
freebsd-386-gce101 is not slow, but we're removing it to shift its
resources to shard other builders wider.
The coordinator now supports varying the build sharding width based on
whether a build is for a trybot or not. This CL defines separate
numbers for each, sharding builds wider as needed for some trybots.
freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down
to 3 when not in try runs.
linux-amd64-race gets one more machine during try runs, and one fewer
in regular runs.
linux-arm goes from 7 machines always, to 3 or 8, depending on whether
it's a try run.
openbsd-amd64-58 goes from 4 to 3 or 6.
windows-amd64-gce goes from 4 to 2 or 6.
windows-amd64-race goes from 4 to 2 or 6.
darwin-amd64-10_11 goes from 3 to 3 or 4.
I'll see how these do over the next few days and readjust as needed.
Also in this CL: fix the constants for the expected duration of
make.bash, which impact when we schedule the creation of test sharding
helper buildlets. We were creating them too early before, wasting
resources.
Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f
Reviewed-on: https://go-review.googlesource.com/29116
Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
|
|
|
Name: "linux-amd64-race",
|
2018-05-14 20:22:45 +03:00
|
|
|
HostType: "host-linux-jessie",
|
2017-04-12 03:35:37 +03:00
|
|
|
TryBot: true,
|
2018-05-04 05:42:17 +03:00
|
|
|
MaxAtOnce: 1,
|
2017-12-07 03:40:40 +03:00
|
|
|
ShouldRunDistTest: fasterTrybots,
|
2018-05-04 05:42:17 +03:00
|
|
|
numTestHelpers: 1,
|
cmd/coordinator, dashboard: remove some trybots, shard others wider
I'm aiming to have trybot runs finish in under 5 minutes.
This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set.
openbsd-386-gce58 is the slowest builder. It has an average speed of
722 seconds (and 95 percentile of 923 seconds) over the past week, and
that's sharded over 4 machines. Too slow. It's not worth the resources
to keep it as a trybot. It hasn't caught any interesting bugs. This
builder will still run, but not as a pre-submit trybot.
freebsd-386-gce101 is not slow, but we're removing it to shift its
resources to shard other builders wider.
The coordinator now supports varying the build sharding width based on
whether a build is for a trybot or not. This CL defines separate
numbers for each, sharding builds wider as needed for some trybots.
freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down
to 3 when not in try runs.
linux-amd64-race gets one more machine during try runs, and one fewer
in regular runs.
linux-arm goes from 7 machines always, to 3 or 8, depending on whether
it's a try run.
openbsd-amd64-58 goes from 4 to 3 or 6.
windows-amd64-gce goes from 4 to 2 or 6.
windows-amd64-race goes from 4 to 2 or 6.
darwin-amd64-10_11 goes from 3 to 3 or 4.
I'll see how these do over the next few days and readjust as needed.
Also in this CL: fix the constants for the expected duration of
make.bash, which impact when we schedule the creation of test sharding
helper buildlets. We were creating them too early before, wasting
resources.
Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f
Reviewed-on: https://go-review.googlesource.com/29116
Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
|
|
|
numTryTestHelpers: 5,
|
2015-01-22 02:15:48 +03:00
|
|
|
})
|
2015-02-13 06:45:02 +03:00
|
|
|
addBuilder(BuildConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Name: "linux-386-clang",
|
|
|
|
|
HostType: "host-linux-clang",
|
|
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
Notes: "Debian jessie + clang 3.9 instead of gcc",
|
|
|
|
|
env: []string{"CC=/usr/bin/clang", "GOHOSTARCH=386"},
|
2015-02-13 06:45:02 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Name: "linux-amd64-clang",
|
|
|
|
|
HostType: "host-linux-clang",
|
|
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
Notes: "Debian jessie + clang 3.9 instead of gcc",
|
|
|
|
|
env: []string{"CC=/usr/bin/clang"},
|
2015-02-13 06:45:02 +03:00
|
|
|
})
|
2015-02-14 06:01:32 +03:00
|
|
|
addBuilder(BuildConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Name: "linux-386-sid",
|
|
|
|
|
HostType: "host-linux-sid",
|
|
|
|
|
Notes: "Debian sid (unstable)",
|
|
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
env: []string{"GOHOSTARCH=386"},
|
2015-02-14 06:01:32 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Name: "linux-amd64-sid",
|
|
|
|
|
HostType: "host-linux-sid",
|
|
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
Notes: "Debian sid (unstable)",
|
2015-02-14 06:01:32 +03:00
|
|
|
})
|
2018-05-14 20:22:45 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "linux-amd64-stretch",
|
|
|
|
|
HostType: "host-linux-stretch",
|
|
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
Notes: "Debian Stretch",
|
|
|
|
|
})
|
2018-05-16 22:50:39 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "linux-amd64-longtest",
|
|
|
|
|
HostType: "host-linux-stretch",
|
|
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
Notes: "Debian Stretch with go test -short=false",
|
2018-05-29 22:03:23 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GO_TEST_SHORT=0",
|
|
|
|
|
// runtime takes ~190 seconds in long mode, which is over
|
|
|
|
|
// the 180 seconds default timeout.
|
|
|
|
|
"GO_TEST_TIMEOUT_SCALE=2",
|
|
|
|
|
},
|
2018-05-16 22:50:39 +03:00
|
|
|
})
|
2015-03-01 20:23:57 +03:00
|
|
|
addBuilder(BuildConfig{
|
cmd/coordinator, dashboard: remove some trybots, shard others wider
I'm aiming to have trybot runs finish in under 5 minutes.
This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set.
openbsd-386-gce58 is the slowest builder. It has an average speed of
722 seconds (and 95 percentile of 923 seconds) over the past week, and
that's sharded over 4 machines. Too slow. It's not worth the resources
to keep it as a trybot. It hasn't caught any interesting bugs. This
builder will still run, but not as a pre-submit trybot.
freebsd-386-gce101 is not slow, but we're removing it to shift its
resources to shard other builders wider.
The coordinator now supports varying the build sharding width based on
whether a build is for a trybot or not. This CL defines separate
numbers for each, sharding builds wider as needed for some trybots.
freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down
to 3 when not in try runs.
linux-amd64-race gets one more machine during try runs, and one fewer
in regular runs.
linux-arm goes from 7 machines always, to 3 or 8, depending on whether
it's a try run.
openbsd-amd64-58 goes from 4 to 3 or 6.
windows-amd64-gce goes from 4 to 2 or 6.
windows-amd64-race goes from 4 to 2 or 6.
darwin-amd64-10_11 goes from 3 to 3 or 4.
I'll see how these do over the next few days and readjust as needed.
Also in this CL: fix the constants for the expected duration of
make.bash, which impact when we schedule the creation of test sharding
helper buildlets. We were creating them too early before, wasting
resources.
Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f
Reviewed-on: https://go-review.googlesource.com/29116
Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
|
|
|
Name: "linux-arm",
|
2017-05-18 00:28:46 +03:00
|
|
|
HostType: "host-linux-arm-scaleway",
|
2017-11-15 23:55:51 +03:00
|
|
|
TryBot: false, // Issue 22748, Issue 22749
|
cmd/coordinator, dashboard: remove some trybots, shard others wider
I'm aiming to have trybot runs finish in under 5 minutes.
This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set.
openbsd-386-gce58 is the slowest builder. It has an average speed of
722 seconds (and 95 percentile of 923 seconds) over the past week, and
that's sharded over 4 machines. Too slow. It's not worth the resources
to keep it as a trybot. It hasn't caught any interesting bugs. This
builder will still run, but not as a pre-submit trybot.
freebsd-386-gce101 is not slow, but we're removing it to shift its
resources to shard other builders wider.
The coordinator now supports varying the build sharding width based on
whether a build is for a trybot or not. This CL defines separate
numbers for each, sharding builds wider as needed for some trybots.
freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down
to 3 when not in try runs.
linux-amd64-race gets one more machine during try runs, and one fewer
in regular runs.
linux-arm goes from 7 machines always, to 3 or 8, depending on whether
it's a try run.
openbsd-amd64-58 goes from 4 to 3 or 6.
windows-amd64-gce goes from 4 to 2 or 6.
windows-amd64-race goes from 4 to 2 or 6.
darwin-amd64-10_11 goes from 3 to 3 or 4.
I'll see how these do over the next few days and readjust as needed.
Also in this CL: fix the constants for the expected duration of
make.bash, which impact when we schedule the creation of test sharding
helper buildlets. We were creating them too early before, wasting
resources.
Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f
Reviewed-on: https://go-review.googlesource.com/29116
Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
|
|
|
FlakyNet: true,
|
|
|
|
|
numTestHelpers: 2,
|
|
|
|
|
numTryTestHelpers: 7,
|
2015-05-14 23:39:58 +03:00
|
|
|
})
|
2015-05-07 21:36:25 +03:00
|
|
|
addBuilder(BuildConfig{
|
2016-09-23 02:29:31 +03:00
|
|
|
Name: "linux-arm-nativemake",
|
|
|
|
|
Notes: "runs make.bash on real ARM hardware, but does not run tests",
|
2017-06-23 17:02:38 +03:00
|
|
|
HostType: "host-linux-arm-scaleway",
|
2016-09-23 02:29:31 +03:00
|
|
|
StopAfterMake: true,
|
|
|
|
|
})
|
2017-03-22 23:53:27 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "linux-arm-arm5spacemonkey",
|
|
|
|
|
HostType: "host-linux-arm5spacemonkey",
|
2017-12-08 20:39:57 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOARM=5",
|
|
|
|
|
"GO_TEST_TIMEOUT_SCALE=4", // arm is normally 2; double that.
|
|
|
|
|
},
|
2017-12-14 21:28:16 +03:00
|
|
|
ShouldRunDistTest: func(distTest string, isTry bool) bool {
|
|
|
|
|
if strings.Contains(distTest, "vendor/github.com/google/pprof") {
|
|
|
|
|
// Not worth it. And broken.
|
|
|
|
|
return false
|
|
|
|
|
}
|
2018-04-10 23:03:26 +03:00
|
|
|
if distTest == "api" {
|
|
|
|
|
// Broken on this build config (Issue
|
|
|
|
|
// 24754), and not worth it on slow
|
|
|
|
|
// builder. It's covered by other
|
|
|
|
|
// builders anyway.
|
|
|
|
|
return false
|
|
|
|
|
}
|
|
|
|
|
if strings.HasPrefix(distTest, "test:") {
|
|
|
|
|
// Slow, and not worth it on slow builder.
|
|
|
|
|
return false
|
|
|
|
|
}
|
2017-12-14 21:28:16 +03:00
|
|
|
return true
|
|
|
|
|
},
|
2017-03-22 23:53:27 +03:00
|
|
|
})
|
2015-02-19 01:12:22 +03:00
|
|
|
addBuilder(BuildConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Name: "nacl-386",
|
|
|
|
|
HostType: "host-nacl-kube",
|
|
|
|
|
TryBot: true,
|
|
|
|
|
MaxAtOnce: 2,
|
|
|
|
|
numTryTestHelpers: 3,
|
|
|
|
|
env: []string{"GOOS=nacl", "GOARCH=386", "GOHOSTOS=linux", "GOHOSTARCH=amd64"},
|
2016-01-27 02:13:47 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2018-05-04 05:42:17 +03:00
|
|
|
Name: "nacl-amd64p32",
|
|
|
|
|
HostType: "host-nacl-kube",
|
|
|
|
|
TryBot: true,
|
|
|
|
|
MaxAtOnce: 2,
|
|
|
|
|
numTryTestHelpers: 3,
|
|
|
|
|
env: []string{"GOOS=nacl", "GOARCH=amd64p32", "GOHOSTOS=linux", "GOHOSTARCH=amd64"},
|
2016-01-27 02:13:47 +03:00
|
|
|
})
|
2018-05-11 08:54:32 +03:00
|
|
|
addBuilder(BuildConfig{
|
2018-06-28 23:30:35 +03:00
|
|
|
Name: "js-wasm",
|
|
|
|
|
HostType: "host-js-wasm",
|
|
|
|
|
TryBot: true,
|
|
|
|
|
ShouldRunDistTest: func(distTest string, isTry bool) bool {
|
|
|
|
|
if isTry {
|
|
|
|
|
if strings.HasPrefix(distTest, "test:") {
|
|
|
|
|
return false
|
|
|
|
|
}
|
|
|
|
|
if strings.Contains(distTest, "/internal/") ||
|
|
|
|
|
strings.Contains(distTest, "vendor/golang.org/x/arch") {
|
|
|
|
|
return false
|
|
|
|
|
}
|
|
|
|
|
switch distTest {
|
|
|
|
|
case "cmd/go", "nolibgcc:crypto/x509":
|
|
|
|
|
return false
|
|
|
|
|
}
|
|
|
|
|
return true
|
|
|
|
|
}
|
|
|
|
|
return true
|
|
|
|
|
},
|
2018-06-25 21:12:11 +03:00
|
|
|
numTryTestHelpers: 4,
|
|
|
|
|
GoDeps: []string{
|
|
|
|
|
"3dced519cbabc213df369d9112206986e62687fa", // first passing commit
|
|
|
|
|
},
|
2018-05-11 08:54:32 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOOS=js", "GOARCH=wasm", "GOHOSTOS=linux", "GOHOSTARCH=amd64",
|
|
|
|
|
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/workdir/go/misc/wasm",
|
|
|
|
|
},
|
|
|
|
|
})
|
2016-12-10 22:10:49 +03:00
|
|
|
addBuilder(BuildConfig{
|
2016-12-15 01:41:25 +03:00
|
|
|
Name: "openbsd-amd64-60",
|
2016-12-10 22:10:49 +03:00
|
|
|
HostType: "host-openbsd-amd64-60",
|
2017-12-07 03:40:40 +03:00
|
|
|
ShouldRunDistTest: noTestDir,
|
2018-02-27 20:26:26 +03:00
|
|
|
TryOnly: true, // disabled by default; Go 1.11+ don't support it anymore
|
2017-11-30 01:43:26 +03:00
|
|
|
TryBot: false,
|
|
|
|
|
MaxAtOnce: 1,
|
2016-12-10 22:10:49 +03:00
|
|
|
numTestHelpers: 2,
|
|
|
|
|
numTryTestHelpers: 5,
|
|
|
|
|
})
|
2015-01-15 23:46:22 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-12-07 03:40:40 +03:00
|
|
|
Name: "openbsd-386-60",
|
|
|
|
|
HostType: "host-openbsd-386-60",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
2018-02-27 20:26:26 +03:00
|
|
|
TryOnly: true, // disabled by default; Go 1.11+ don't support it anymore
|
|
|
|
|
TryBot: false,
|
2017-12-07 03:40:40 +03:00
|
|
|
MaxAtOnce: 1,
|
2017-12-14 21:28:16 +03:00
|
|
|
env: []string{
|
|
|
|
|
// cmd/go takes ~192 seconds on openbsd-386
|
|
|
|
|
// now, which is over the 180 second default
|
|
|
|
|
// dist test timeout. So, bump this builder
|
|
|
|
|
// up:
|
|
|
|
|
"GO_TEST_TIMEOUT_SCALE=2",
|
|
|
|
|
},
|
2015-01-15 23:46:22 +03:00
|
|
|
})
|
2017-12-05 08:22:36 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-12-07 03:40:40 +03:00
|
|
|
Name: "openbsd-386-62",
|
|
|
|
|
HostType: "host-openbsd-386-62",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
|
|
|
|
MaxAtOnce: 1,
|
2017-12-14 21:28:16 +03:00
|
|
|
env: []string{
|
|
|
|
|
// cmd/go takes ~192 seconds on openbsd-386
|
|
|
|
|
// now, which is over the 180 second default
|
|
|
|
|
// dist test timeout. So, bump this builder
|
|
|
|
|
// up:
|
|
|
|
|
"GO_TEST_TIMEOUT_SCALE=2",
|
|
|
|
|
},
|
2017-12-05 08:22:36 +03:00
|
|
|
})
|
2017-11-24 21:14:33 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "openbsd-amd64-62",
|
|
|
|
|
HostType: "host-openbsd-amd64-62",
|
2017-12-07 03:40:40 +03:00
|
|
|
ShouldRunDistTest: noTestDir,
|
2017-11-30 01:43:26 +03:00
|
|
|
TryBot: true,
|
2017-11-24 21:14:33 +03:00
|
|
|
numTestHelpers: 0,
|
|
|
|
|
numTryTestHelpers: 5,
|
2017-11-30 01:43:26 +03:00
|
|
|
MaxAtOnce: 1,
|
2017-11-24 21:14:33 +03:00
|
|
|
})
|
2018-10-05 01:14:50 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "openbsd-amd64-64",
|
|
|
|
|
HostType: "host-openbsd-amd64-64",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
|
|
|
|
TryBot: false,
|
|
|
|
|
numTestHelpers: 0,
|
|
|
|
|
numTryTestHelpers: 5,
|
|
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "openbsd-386-64",
|
|
|
|
|
HostType: "host-openbsd-386-64",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
|
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
})
|
2016-04-04 21:03:49 +03:00
|
|
|
addBuilder(BuildConfig{
|
2018-04-28 01:47:56 +03:00
|
|
|
Name: "netbsd-amd64-8_0",
|
|
|
|
|
HostType: "host-netbsd-amd64-8_0",
|
2017-12-07 03:40:40 +03:00
|
|
|
ShouldRunDistTest: noTestDir,
|
|
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
TryBot: false,
|
2017-02-26 21:11:21 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2018-04-28 01:47:56 +03:00
|
|
|
Name: "netbsd-386-8_0",
|
|
|
|
|
HostType: "host-netbsd-386-8_0",
|
2017-12-07 03:40:40 +03:00
|
|
|
ShouldRunDistTest: noTestDir,
|
|
|
|
|
MaxAtOnce: 1,
|
2018-04-28 01:47:56 +03:00
|
|
|
// This builder currently hangs in the “../test” phase of all.bash.
|
|
|
|
|
// (https://golang.org/issue/25206)
|
|
|
|
|
TryOnly: true, // Disable regular builds.
|
|
|
|
|
TryBot: false, // Disable trybots.
|
2016-04-04 21:03:49 +03:00
|
|
|
})
|
2015-01-15 23:46:22 +03:00
|
|
|
addBuilder(BuildConfig{
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Name: "plan9-386",
|
|
|
|
|
HostType: "host-plan9-386-gce",
|
cmd/coordinator, dashboard: remove some trybots, shard others wider
I'm aiming to have trybot runs finish in under 5 minutes.
This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set.
openbsd-386-gce58 is the slowest builder. It has an average speed of
722 seconds (and 95 percentile of 923 seconds) over the past week, and
that's sharded over 4 machines. Too slow. It's not worth the resources
to keep it as a trybot. It hasn't caught any interesting bugs. This
builder will still run, but not as a pre-submit trybot.
freebsd-386-gce101 is not slow, but we're removing it to shift its
resources to shard other builders wider.
The coordinator now supports varying the build sharding width based on
whether a build is for a trybot or not. This CL defines separate
numbers for each, sharding builds wider as needed for some trybots.
freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down
to 3 when not in try runs.
linux-amd64-race gets one more machine during try runs, and one fewer
in regular runs.
linux-arm goes from 7 machines always, to 3 or 8, depending on whether
it's a try run.
openbsd-amd64-58 goes from 4 to 3 or 6.
windows-amd64-gce goes from 4 to 2 or 6.
windows-amd64-race goes from 4 to 2 or 6.
darwin-amd64-10_11 goes from 3 to 3 or 4.
I'll see how these do over the next few days and readjust as needed.
Also in this CL: fix the constants for the expected duration of
make.bash, which impact when we schedule the creation of test sharding
helper buildlets. We were creating them too early before, wasting
resources.
Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f
Reviewed-on: https://go-review.googlesource.com/29116
Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
|
|
|
numTestHelpers: 1,
|
2017-11-30 01:43:26 +03:00
|
|
|
MaxAtOnce: 2,
|
2015-01-15 23:46:22 +03:00
|
|
|
})
|
2017-04-21 22:35:28 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-12-07 03:40:40 +03:00
|
|
|
Name: "windows-amd64-2008",
|
|
|
|
|
HostType: "host-windows-amd64-2008",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
2018-05-01 00:16:33 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOARCH=amd64",
|
|
|
|
|
"GOHOSTARCH=amd64",
|
|
|
|
|
// cmd/go takes ~188 seconds on windows-amd64
|
|
|
|
|
// now, which is over the 180 second default
|
|
|
|
|
// dist test timeout. So, bump this builder
|
|
|
|
|
// up:
|
|
|
|
|
"GO_TEST_TIMEOUT_SCALE=2",
|
|
|
|
|
},
|
2017-04-21 22:35:28 +03:00
|
|
|
})
|
2017-04-27 23:12:08 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "windows-386-2008",
|
|
|
|
|
HostType: "host-windows-amd64-2008",
|
2017-12-07 03:40:40 +03:00
|
|
|
ShouldRunDistTest: fasterTrybots,
|
2017-04-27 23:12:08 +03:00
|
|
|
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
|
2017-11-30 01:43:26 +03:00
|
|
|
MaxAtOnce: 2,
|
2017-07-27 07:52:12 +03:00
|
|
|
TryBot: true,
|
2017-12-07 03:40:40 +03:00
|
|
|
numTryTestHelpers: 4,
|
2017-04-27 23:12:08 +03:00
|
|
|
})
|
2017-04-21 22:35:28 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-12-07 03:40:40 +03:00
|
|
|
Name: "windows-amd64-2012",
|
|
|
|
|
HostType: "host-windows-amd64-2012",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
2018-05-01 00:16:33 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOARCH=amd64",
|
|
|
|
|
"GOHOSTARCH=amd64",
|
|
|
|
|
// cmd/go takes ~188 seconds on windows-amd64
|
|
|
|
|
// now, which is over the 180 second default
|
|
|
|
|
// dist test timeout. So, bump this builder
|
|
|
|
|
// up:
|
|
|
|
|
"GO_TEST_TIMEOUT_SCALE=2",
|
|
|
|
|
},
|
|
|
|
|
MaxAtOnce: 2,
|
2017-04-21 22:35:28 +03:00
|
|
|
})
|
2015-02-07 04:32:15 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-07-27 07:52:12 +03:00
|
|
|
Name: "windows-amd64-2016",
|
|
|
|
|
HostType: "host-windows-amd64-2016",
|
2017-12-07 03:40:40 +03:00
|
|
|
ShouldRunDistTest: fasterTrybots,
|
2018-05-01 00:16:33 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOARCH=amd64",
|
|
|
|
|
"GOHOSTARCH=amd64",
|
|
|
|
|
// cmd/go takes ~188 seconds on windows-amd64
|
|
|
|
|
// now, which is over the 180 second default
|
|
|
|
|
// dist test timeout. So, bump this builder
|
|
|
|
|
// up:
|
|
|
|
|
"GO_TEST_TIMEOUT_SCALE=2",
|
|
|
|
|
},
|
2017-07-27 07:52:12 +03:00
|
|
|
TryBot: true,
|
|
|
|
|
numTryTestHelpers: 5,
|
2015-02-07 04:32:15 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Name: "windows-amd64-race",
|
2017-04-24 23:32:02 +03:00
|
|
|
HostType: "host-windows-amd64-2008",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Notes: "Only runs -race tests (./race.bat)",
|
2018-05-01 00:16:33 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOARCH=amd64",
|
|
|
|
|
"GOHOSTARCH=amd64",
|
|
|
|
|
// cmd/go takes ~188 seconds on windows-amd64
|
|
|
|
|
// now, which is over the 180 second default
|
|
|
|
|
// dist test timeout. So, bump this builder
|
|
|
|
|
// up:
|
|
|
|
|
"GO_TEST_TIMEOUT_SCALE=2"},
|
2015-02-07 04:32:15 +03:00
|
|
|
})
|
2018-07-24 22:59:34 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "windows-arm",
|
|
|
|
|
HostType: "host-windows-arm-iotcore",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOARM=7",
|
|
|
|
|
"GO_TEST_TIMEOUT_SCALE=2",
|
|
|
|
|
},
|
|
|
|
|
})
|
2016-08-30 22:55:08 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-12-07 03:40:40 +03:00
|
|
|
Name: "darwin-amd64-10_8",
|
|
|
|
|
HostType: "host-darwin-10_8",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
2018-02-20 22:36:16 +03:00
|
|
|
TryOnly: true, // but not in trybot set, so effectively disabled
|
|
|
|
|
TryBot: false,
|
2016-08-30 22:55:08 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2017-12-07 03:40:40 +03:00
|
|
|
Name: "darwin-amd64-10_10",
|
|
|
|
|
HostType: "host-darwin-10_10",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
2016-08-30 22:55:08 +03:00
|
|
|
})
|
2016-09-07 08:41:02 +03:00
|
|
|
addBuilder(BuildConfig{
|
2018-02-16 02:43:31 +03:00
|
|
|
Name: "darwin-amd64-10_11",
|
|
|
|
|
HostType: "host-darwin-10_11",
|
|
|
|
|
TryBot: false, // disabled until Macs fixed; https://golang.org/issue/23859
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
|
|
|
|
numTryTestHelpers: 3,
|
2016-09-07 08:41:02 +03:00
|
|
|
})
|
2017-04-25 02:03:42 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-12-07 03:40:40 +03:00
|
|
|
Name: "darwin-386-10_11",
|
|
|
|
|
HostType: "host-darwin-10_11",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
|
|
|
|
MaxAtOnce: 1,
|
|
|
|
|
env: []string{"GOARCH=386", "GOHOSTARCH=386"},
|
|
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "darwin-amd64-10_12",
|
|
|
|
|
HostType: "host-darwin-10_12",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
2017-04-25 02:03:42 +03:00
|
|
|
})
|
2017-01-19 01:45:12 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-12-07 03:40:40 +03:00
|
|
|
Name: "darwin-amd64-race",
|
|
|
|
|
HostType: "host-darwin-10_11",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
2017-01-19 01:45:12 +03:00
|
|
|
})
|
2017-02-02 18:00:21 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-09-29 15:10:42 +03:00
|
|
|
Name: "darwin-arm-a1428ios",
|
2017-08-28 20:11:57 +03:00
|
|
|
HostType: "host-darwin-amd64-eliasnaur-ios",
|
2018-06-18 16:50:26 +03:00
|
|
|
Notes: "iPhone 6 (model A1586), via a Mac Mini; owned by elias.naur",
|
|
|
|
|
env: []string{"GOARCH=arm"},
|
2017-02-02 18:00:21 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2017-09-29 15:10:42 +03:00
|
|
|
Name: "darwin-arm64-a1549ios",
|
2017-08-28 20:11:57 +03:00
|
|
|
HostType: "host-darwin-amd64-eliasnaur-ios",
|
2018-06-18 16:50:26 +03:00
|
|
|
Notes: "iPhone 6 (model A1586), via a Mac Mini; owned by elias.naur",
|
|
|
|
|
env: []string{"GOARCH=arm64"},
|
2017-02-02 18:00:21 +03:00
|
|
|
})
|
2018-07-25 16:14:32 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "darwin-arm-mg912baios",
|
|
|
|
|
HostType: "host-darwin-amd64-zenly-ios",
|
|
|
|
|
Notes: "iPhone 5C (model MG912B/A), via a MacBook Pro; owned by zenly",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOARCH=arm",
|
|
|
|
|
"GOIOS_DEVICE_ID=8e5c23a5d0843d1ffe164ea0b2f2500599c3ebff",
|
|
|
|
|
},
|
|
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "darwin-arm64-mn4m2zdaios",
|
|
|
|
|
HostType: "host-darwin-amd64-zenly-ios",
|
|
|
|
|
Notes: "iPhone 7+ (model MN4M2ZD/A), via a MacBook Pro; owned by zenly",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOARCH=arm64",
|
|
|
|
|
"GOIOS_DEVICE_ID=5ec20fafe317e1c8ff51efc6d508cf19808474a2",
|
|
|
|
|
},
|
|
|
|
|
})
|
2017-03-30 00:23:59 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "android-arm-wiko-fever",
|
2017-03-31 11:08:01 +03:00
|
|
|
HostType: "host-darwin-amd64-eliasnaur-android",
|
2017-03-30 00:23:59 +03:00
|
|
|
Notes: "Android Wiko Fever phone running Android 6.0, via a Mac Mini",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOARCH=arm",
|
|
|
|
|
"GOARM=7",
|
2018-01-20 20:47:20 +03:00
|
|
|
"GOANDROID_ADB_FLAGS=-d", // Run on device
|
2017-04-02 17:59:34 +03:00
|
|
|
"CC_FOR_TARGET=/Users/elias/android-ndk-standalone-arm/bin/clang",
|
2017-03-30 00:23:59 +03:00
|
|
|
},
|
|
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "android-arm64-wiko-fever",
|
2017-03-31 11:08:01 +03:00
|
|
|
HostType: "host-darwin-amd64-eliasnaur-android",
|
2017-03-30 00:23:59 +03:00
|
|
|
Notes: "Android Wiko Fever phone running Android 6.0, via a Mac Mini",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOARCH=arm64",
|
2018-01-20 20:47:20 +03:00
|
|
|
"GOANDROID_ADB_FLAGS=-d", // Run on device
|
2017-04-02 17:59:34 +03:00
|
|
|
"CC_FOR_TARGET=/Users/elias/android-ndk-standalone-arm64/bin/clang",
|
2017-03-30 00:23:59 +03:00
|
|
|
},
|
|
|
|
|
})
|
2018-01-20 20:47:20 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "android-386-emulator",
|
|
|
|
|
HostType: "host-darwin-amd64-eliasnaur-android",
|
|
|
|
|
Notes: "Android emulator, via a Mac Mini",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOARCH=386",
|
|
|
|
|
"GOANDROID_ADB_FLAGS=-e", // Run on emulator
|
|
|
|
|
"CC_FOR_TARGET=/Users/elias/android-ndk-standalone-386/bin/clang",
|
|
|
|
|
},
|
|
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "android-amd64-emulator",
|
|
|
|
|
HostType: "host-darwin-amd64-eliasnaur-android",
|
|
|
|
|
Notes: "Android emulator, via a Mac Mini",
|
|
|
|
|
env: []string{
|
|
|
|
|
"GOARCH=amd64",
|
|
|
|
|
"GOANDROID_ADB_FLAGS=-e", // Run on emulator
|
|
|
|
|
"CC_FOR_TARGET=/Users/elias/android-ndk-standalone-amd64/bin/clang",
|
|
|
|
|
},
|
|
|
|
|
})
|
2017-06-27 20:11:15 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "solaris-amd64-oracledev",
|
|
|
|
|
HostType: "host-solaris-oracle-shawn",
|
|
|
|
|
Notes: "Oracle Solaris development version",
|
|
|
|
|
})
|
2017-07-07 23:41:20 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "solaris-amd64-oraclerel",
|
|
|
|
|
HostType: "host-solaris-oracle-amd64-oraclerel",
|
|
|
|
|
Notes: "Oracle Solaris release version",
|
|
|
|
|
})
|
2016-03-01 07:49:24 +03:00
|
|
|
addBuilder(BuildConfig{
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Name: "solaris-amd64-smartosbuildlet",
|
|
|
|
|
HostType: "host-solaris-amd64",
|
2016-03-01 07:49:24 +03:00
|
|
|
})
|
2016-10-06 23:30:58 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "linux-ppc64-buildlet",
|
|
|
|
|
HostType: "host-linux-ppc64-osu",
|
|
|
|
|
FlakyNet: true,
|
|
|
|
|
})
|
2016-03-01 07:49:24 +03:00
|
|
|
addBuilder(BuildConfig{
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Name: "linux-ppc64le-buildlet",
|
|
|
|
|
HostType: "host-linux-ppc64le-osu",
|
|
|
|
|
FlakyNet: true,
|
2016-04-07 22:27:15 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Name: "linux-arm64-buildlet",
|
|
|
|
|
HostType: "host-linux-arm64-linaro",
|
|
|
|
|
FlakyNet: true,
|
2016-03-01 07:49:24 +03:00
|
|
|
})
|
2017-04-07 01:16:28 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "linux-arm64-packet",
|
|
|
|
|
HostType: "host-linux-arm64-packet",
|
|
|
|
|
FlakyNet: true, // unknown; just copied from the linaro one
|
|
|
|
|
})
|
2016-11-03 20:35:15 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-04-13 07:34:42 +03:00
|
|
|
Name: "linux-mips",
|
|
|
|
|
HostType: "host-linux-mips",
|
|
|
|
|
SkipSnapshot: true,
|
2016-11-03 20:35:15 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2017-04-13 07:34:42 +03:00
|
|
|
Name: "linux-mipsle",
|
|
|
|
|
HostType: "host-linux-mipsle",
|
|
|
|
|
SkipSnapshot: true,
|
2016-11-03 20:35:15 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2017-04-13 07:34:42 +03:00
|
|
|
Name: "linux-mips64",
|
|
|
|
|
HostType: "host-linux-mips64",
|
|
|
|
|
SkipSnapshot: true,
|
2016-11-03 20:35:15 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2017-04-13 07:34:42 +03:00
|
|
|
Name: "linux-mips64le",
|
|
|
|
|
HostType: "host-linux-mips64le",
|
|
|
|
|
SkipSnapshot: true,
|
2016-11-03 20:35:15 +03:00
|
|
|
})
|
2016-03-01 07:49:24 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "linux-s390x-ibm",
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
HostType: "host-linux-s390x",
|
cmd/coordinator, dashboard: remove some trybots, shard others wider
I'm aiming to have trybot runs finish in under 5 minutes.
This CL removes openbsd-386-gce58 and freebsd-386-gce101 from the trybot set.
openbsd-386-gce58 is the slowest builder. It has an average speed of
722 seconds (and 95 percentile of 923 seconds) over the past week, and
that's sharded over 4 machines. Too slow. It's not worth the resources
to keep it as a trybot. It hasn't caught any interesting bugs. This
builder will still run, but not as a pre-submit trybot.
freebsd-386-gce101 is not slow, but we're removing it to shift its
resources to shard other builders wider.
The coordinator now supports varying the build sharding width based on
whether a build is for a trybot or not. This CL defines separate
numbers for each, sharding builds wider as needed for some trybots.
freebsd-amd64-gce101 goes from 4 to 5 machines in try runs, and down
to 3 when not in try runs.
linux-amd64-race gets one more machine during try runs, and one fewer
in regular runs.
linux-arm goes from 7 machines always, to 3 or 8, depending on whether
it's a try run.
openbsd-amd64-58 goes from 4 to 3 or 6.
windows-amd64-gce goes from 4 to 2 or 6.
windows-amd64-race goes from 4 to 2 or 6.
darwin-amd64-10_11 goes from 3 to 3 or 4.
I'll see how these do over the next few days and readjust as needed.
Also in this CL: fix the constants for the expected duration of
make.bash, which impact when we schedule the creation of test sharding
helper buildlets. We were creating them too early before, wasting
resources.
Change-Id: I38a9b24841e196f1eb668de058c49af8c1d1c64f
Reviewed-on: https://go-review.googlesource.com/29116
Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-14 01:45:48 +03:00
|
|
|
numTestHelpers: 0,
|
2016-03-01 07:49:24 +03:00
|
|
|
})
|
2016-08-30 23:32:33 +03:00
|
|
|
addBuilder(BuildConfig{
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
Name: "linux-s390x-crosscompile",
|
|
|
|
|
HostType: "host-s390x-cross-kube",
|
|
|
|
|
Notes: "s390x cross-compile builder for releases; doesn't run tests",
|
|
|
|
|
CompileOnly: true,
|
|
|
|
|
TryOnly: true, // but not in trybot set for now
|
2016-08-30 23:32:33 +03:00
|
|
|
env: []string{
|
|
|
|
|
"CGO_ENABLED=1",
|
|
|
|
|
"GOARCH=s390x",
|
|
|
|
|
"GOHOSTARCH=amd64",
|
|
|
|
|
"CC_FOR_TARGET=s390x-linux-gnu-gcc",
|
|
|
|
|
},
|
|
|
|
|
})
|
cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixes golang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-22 22:15:56 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "linux-amd64-localdev",
|
|
|
|
|
HostType: "host-linux-amd64-localdev",
|
|
|
|
|
Notes: "for localhost development only",
|
|
|
|
|
TryOnly: true,
|
|
|
|
|
})
|
2017-08-06 01:56:18 +03:00
|
|
|
addBuilder(BuildConfig{
|
2017-12-07 03:40:40 +03:00
|
|
|
Name: "dragonfly-amd64",
|
|
|
|
|
HostType: "host-dragonfly-amd64-tdfbsd",
|
|
|
|
|
ShouldRunDistTest: noTestDir,
|
|
|
|
|
SkipSnapshot: true,
|
2017-08-06 01:56:18 +03:00
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
2017-08-06 06:54:04 +03:00
|
|
|
Name: "freebsd-arm-paulzhol",
|
2017-08-06 02:58:12 +03:00
|
|
|
HostType: "host-freebsd-arm-paulzhol",
|
|
|
|
|
SkipSnapshot: true,
|
2017-08-06 01:56:18 +03:00
|
|
|
env: []string{
|
|
|
|
|
"GOARM=7",
|
|
|
|
|
"CGO_ENABLED=1",
|
|
|
|
|
},
|
|
|
|
|
})
|
2017-08-06 18:57:06 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "plan9-arm",
|
|
|
|
|
HostType: "host-plan9-arm-0intro",
|
|
|
|
|
SkipSnapshot: true,
|
|
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "nacl-arm",
|
|
|
|
|
HostType: "host-nacl-arm-davecheney",
|
|
|
|
|
SkipSnapshot: true,
|
|
|
|
|
})
|
|
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "plan9-amd64-9front",
|
|
|
|
|
HostType: "host-plan9-amd64-0intro",
|
|
|
|
|
SkipSnapshot: true,
|
|
|
|
|
})
|
2018-08-24 21:41:05 +03:00
|
|
|
addBuilder(BuildConfig{
|
|
|
|
|
Name: "aix-ppc64",
|
|
|
|
|
HostType: "host-aix-ppc64-osuosl",
|
|
|
|
|
})
|
2017-08-06 01:56:18 +03:00
|
|
|
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
}
|
2016-08-30 23:32:33 +03:00
|
|
|
|
2017-04-12 03:35:37 +03:00
|
|
|
// addBuilder adds c to the Builders map after doing some sanity
|
|
|
|
|
// checks.
|
2015-01-15 23:46:22 +03:00
|
|
|
func addBuilder(c BuildConfig) {
|
|
|
|
|
if c.Name == "" {
|
|
|
|
|
panic("empty name")
|
|
|
|
|
}
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
if c.HostType == "" {
|
|
|
|
|
panic(fmt.Sprintf("missing HostType for builder %q", c.Name))
|
|
|
|
|
}
|
2015-01-15 23:46:22 +03:00
|
|
|
if _, dup := Builders[c.Name]; dup {
|
2017-08-06 18:57:06 +03:00
|
|
|
panic("dup name " + c.Name)
|
2015-01-15 23:46:22 +03:00
|
|
|
}
|
all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".
Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.
Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):
linux-amd64
linux-amd64-race
linux-386
linux-386-387
And these are the same host type (a GCE VM):
windows-amd64-gce
windows-amd64-race
windows-386-gce
This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.
Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.
Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.
gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.
This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.
Updates golang/go#17104
Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 00:27:37 +03:00
|
|
|
if _, ok := Hosts[c.HostType]; !ok {
|
|
|
|
|
panic(fmt.Sprintf("undefined HostType %q for builder %q", c.HostType, c.Name))
|
2015-09-09 01:18:47 +03:00
|
|
|
}
|
2017-04-13 07:34:42 +03:00
|
|
|
if c.SkipSnapshot && (c.numTestHelpers > 0 || c.numTryTestHelpers > 0) {
|
|
|
|
|
panic(fmt.Sprintf("config %q's SkipSnapshot is not compatible with sharded test helpers", c.Name))
|
|
|
|
|
}
|
2017-04-12 03:35:37 +03:00
|
|
|
|
|
|
|
|
types := 0
|
2018-05-05 19:36:05 +03:00
|
|
|
for _, fn := range []func() bool{c.IsReverse, c.IsContainer, c.IsVM} {
|
2017-04-12 03:35:37 +03:00
|
|
|
if fn() {
|
|
|
|
|
types++
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
if types != 1 {
|
2018-05-04 05:42:17 +03:00
|
|
|
panic(fmt.Sprintf("build config %q host type inconsistent (must be Reverse, Image, or VM)", c.Name))
|
2017-04-12 03:35:37 +03:00
|
|
|
}
|
|
|
|
|
|
2018-10-26 20:52:25 +03:00
|
|
|
Builders[c.Name] = &c
|
2015-01-15 23:46:22 +03:00
|
|
|
}
|
2017-04-12 03:35:37 +03:00
|
|
|
|
|
|
|
|
// TrybotBuilderNames returns the names of the builder configs
|
|
|
|
|
// with the TryBot field set true.
|
|
|
|
|
func TrybotBuilderNames() []string {
|
|
|
|
|
var ret []string
|
|
|
|
|
for name, conf := range Builders {
|
|
|
|
|
if conf.TryBot {
|
|
|
|
|
ret = append(ret, name)
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
sort.Strings(ret)
|
|
|
|
|
return ret
|
|
|
|
|
}
|
2017-12-07 03:40:40 +03:00
|
|
|
|
|
|
|
|
// fasterTrybots is a ShouldRunDistTest policy function.
|
|
|
|
|
// It skips (returns false) the test/ directory tests for trybots.
|
|
|
|
|
func fasterTrybots(distTest string, isTry bool) bool {
|
|
|
|
|
if isTry && strings.HasPrefix(distTest, "test:") {
|
|
|
|
|
return false // skip test
|
|
|
|
|
}
|
|
|
|
|
return true
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// noTestDir is a ShouldRunDistTest policy function.
|
|
|
|
|
// It skips (returns false) the test/ directory tests for all builds.
|
|
|
|
|
func noTestDir(distTest string, isTry bool) bool {
|
|
|
|
|
if strings.HasPrefix(distTest, "test:") {
|
|
|
|
|
return false // skip test
|
|
|
|
|
}
|
|
|
|
|
return true
|
|
|
|
|
}
|
