Граф коммитов

7 Коммитов

Автор SHA1 Сообщение Дата
Brad Fitzpatrick 6ae8750e84 all: use Container-Optimized VMs instead of Kubernetes for buildlet containers
Fixes golang/go#25108

Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9
Reviewed-on: https://go-review.googlesource.com/111267
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-11 03:29:07 +00:00
Brad Fitzpatrick 4eceee2d0f cmd/coordinator, cmd/buildlet, cmd/gomote: add SSH support
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.

For example:

    $ gomote create openbsd-amd64-60
    user-bradfitz-openbsd-amd64-60-1

    $ gomote ssh user-bradfitz-openbsd-amd64-60-1
    Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
    OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016

    Welcome to OpenBSD: The proactively secure Unix-like operating system.

    Please use the sendbug(1) utility to report bugs in the system.
    Before reporting a bug, please try to reproduce it with the latest
    version of the code.  With bug reports, please try to ensure that
    enough information to reproduce the problem is enclosed, and if a
    known fix for it exists, include that as well.

    $

As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.

Not yet supported:

* scp (help wanted)
* not all host types are configured. most are. some will need slight
  config tweaks to the Docker image (e.g. adding openssh-server)

Supports currently:

* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows

Implementation details:

* the ssh server process listens on port 2222 in the coordinator
  (farmer.golang.org), which is behind a GKE TCP load balancer.

* the ssh server library is github.com/gliderlabs/ssh

* authentication is done via Github users' public keys. It's assumed
  that gomote user == github user. But there's a mapping in the code
  for known exceptions.

* we can't give out access to this too widely. too many things are
  accessible from within the host environment if you look in the right
  places. Details omitted. But the Go team and other trusted gomote
  users can use this.

* the buildlet binary has a new /connect-ssh handler that acts like a
  CONNECT request but instead of taking an explicit host:port, just
  says "give me your machine's SSH connection". The buildlet can also
  start sshd if needed for the environment. The /connect-ssh handler
  also installs the coordinator's public key.

* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
  handler and returns a net.Conn.

* the coordinator's ssh.Handler is just running the OpenSSH ssh client.

* because the OpenSSH ssh child process can't connect to a net.Conn,
  an emphemeral localhost port is created on the coordinator to proxy
  between the ssh client and the net.Conn returned by ConnectSSH.

* The /connect-ssh handler requires http.Hijacker, which requires
  fully compliant net.Conn implementations as of Go 1.8. So I needed
  to flesh out revdial too, testing it with the
  golang.org/x/net/nettest package.

* plan9 doesn't have an ssh server, so we use 0intro's new conterm
  program (drawterm without GUI support) to connect to plan9 from the
  coordinator ssh proxy instead of using the OpenSSH ssh client
  binary.

* windows doesn't have an ssh server, so we enable the telnet service
  and the coordinator ssh proxy uses telnet instead on the backend
  on the private network. (There is a Windows ssh server but only in
  new versions.)

Happy debugging over ssh!

Fixes golang/go#19956

Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-07-28 18:21:11 +00:00
Josh Bleecher Snyder d5f7ed3f5b buildlet: add config file to alter username
My gomote username is josharian.
My whoami is josh.
As a result, I need to run 'gomote -user=josharian'
every time, which is an annoyance.

Add a simple config file to allow a user to change their user name.
On startup, gomote checks whether user-$(whoami).user exists.
If so, it reads it to get the real username.
So I have a config file ~/.config/gomote/user-josh.user
with the contents 'josharian'.

It's a bit of an ugly hack, but it works and spares me some pain.

Change-Id: I372d5a786b99c9e3c6a57f25b0a38b9146f23598
Reviewed-on: https://go-review.googlesource.com/42230
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-05-01 18:04:27 +00:00
Brad Fitzpatrick e4d08a5aad cmd/gomote, buildenv, buildlet: move config code to common places
Split off from Quentin's https://golang.org/cl/29399

Change-Id: I4578f8f485e97d6b9844fb12e84779167755752e
Reviewed-on: https://go-review.googlesource.com/29858
Reviewed-by: Quentin Smith <quentin@golang.org>
2016-09-27 18:47:08 +00:00
Brad Fitzpatrick 328c9b8918 all: split builder config into builder & host configs
Our builders are named of the form "GOOS-GOARCH" or
"GOOS-GOARCH-suffix".

Over time we've grown many builders. This CL doesn't change
that. Builders continue to be named and operate as before.

Previously the build configuration file (dashboard/builders.go) made
each builder type ("linux-amd64-race", etc) define how to create a
host running a buildlet of that type, even though many builders had
identical host configs. For example, these builders all share the same
host type (a Kubernetes container):

   linux-amd64
   linux-amd64-race
   linux-386
   linux-386-387

And these are the same host type (a GCE VM):

   windows-amd64-gce
   windows-amd64-race
   windows-386-gce

This CL creates a new concept of a "hostType" which defines how
the buildlet is created (Kube, GCE, Reverse, and how), and then each
builder itself references a host type.

Users never see the hostType. (except perhaps in gomote list output)
But they at least never need to care about them.

Reverse buildlets now can only be one hostType at a time, which
simplifies things. We were no longer using multiple roles per machine
once moving to VMs for OS X.

gomote continues to operate as it did previously but its underlying
protocol changed and clients will need to be updated. As a new
feature, gomote now has a new flag to let you reuse a buildlet host
connection for different builder rules if they share the same
underlying host type. But users can ignore that.

This CL is a long-standing TODO (previously attempted and aborted) and
will make many things easier and faster, including the linux-arm
cross-compilation effort, and keeping pre-warmed buildlets of VM types
ready to go.

Updates golang/go#17104

Change-Id: Iad8387f48680424a8441e878a2f4762bf79ea4d2
Reviewed-on: https://go-review.googlesource.com/29551
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2016-09-22 23:13:34 +00:00
Matthew Dempsky d5f422f474 all: fix vet warnings
Change-Id: Ia4519d9b050136b4d46dc8361f546e3c4205a9cd
Reviewed-on: https://go-review.googlesource.com/16288
Reviewed-by: Dave Cheney <dave@cheney.net>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-10-25 04:14:39 +00:00
Brad Fitzpatrick 7b6d1b1b28 all: remote buildlets
This creates a mechanism for clients (such as cmd/release and
cmd/gomote) to obtain buildlets via the coordinator. Previously
cmd/release and cmd/gomote could only create GCE VMs themselves, and
required the GCE project's credentials. In addition to the awkwardness
of needing to hand out the GCE credentials, it also meant ARM and
Darwin buildlets (which use the reverse buildlet pool) weren't usable.

Instead, this creates a new auth mechanism where the coordinator is
contacted over TLS with key pinning (the CA system isn't used) in the
same way that the reverse builders already dialed into the
coordinator, and then a "user build type" and hash are sent as the
username and password. The same master key is used to sign user
builder keys, and they always start with "user-". (which isn't a GOOS).

Then the coordinator provides an API to create and list buildlets.
They auto-expire after a duration and are auto-renewed upon use.

The buildlet library (as used by cmd/release etc) then proxies HTTP
requests via the coordinator to the backend buildlet.

See doc/remote-buildlet.txt for protocol details.

Change-Id: I12e27eae788fdd91927cb182b950893dc759f8e9
Reviewed-on: https://go-review.googlesource.com/11901
Reviewed-by: Andrew Gerrand <adg@golang.org>
2015-07-07 16:45:21 +00:00