The n2 machine types that we now use for instances are either Ice Lake
or Cascade Lake. The error message we received when creating a VM was
specifically to use Cascade Lake, not the older Haswell.
For golang/go#48969
Change-Id: I1c05679c2b512ca3bb7dbb0d2efa6f587305f6aa
Reviewed-on: https://go-review.googlesource.com/c/build/+/356629
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
We should have similar performance for these machines, and increased
memory availability in some areas.
This is also compliant with an internal policy change.
Change-Id: I07a452a07431322d6fdd4edbb98c5914045d34ed
Reviewed-on: https://go-review.googlesource.com/c/build/+/354757
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Resubmission of https://golang.org/cl/354642 now that we have our NAT
config sorted. Hopefully.
We're hitting our external IP quota. Now that we have Cloud NAT set up,
we don't need the external IPs at all. Disable them and stop checking
for them.
Change-Id: I5091ebacd79eb0bddec394f35eb9c7368a699ac4
Reviewed-on: https://go-review.googlesource.com/c/build/+/354756
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
This reverts commit 7e966852b8 (https://golang.org/cl/354642).
Reason for revert: breaks internet access in -longtest builders somehow
Change-Id: I5344e0c8cec5bf93a23c2c3bf592215843e129de
Reviewed-on: https://go-review.googlesource.com/c/build/+/354753
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
We're hitting our external IP quota. Now that we have Cloud NAT set up,
we don't need the external IPs at all. Disable them and stop checking
for them.
(I don't understand why the code wanted to use an external IP when TLS is
enabled, so I ripped it out. If there's a reason I'm missing let me
know.)
Change-Id: I3e66b2ed984d936aa0f58d3e7178ab7356d73ce1
Reviewed-on: https://go-review.googlesource.com/c/build/+/354642
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
The a1.metal instances take about four minutes to become available
before beginning to boot the Windows ARM64 VM. This means we need
somewhere between five and ten minutes before we can first connect to
the buildlet.
It's possible that other instances are faster to boot, but this is
currently the most economical choice.
For golang/go#42604
Change-Id: Ib76944307c9ad28fd1adca0dc54f3fbeaabfffa0
Reviewed-on: https://go-review.googlesource.com/c/build/+/322654
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
update-readmes.go is updated to add a pkg.go.dev badge instead of a
godoc.org badge.
The GoDoc badge is added or changed in all READMEs by running
update-readmes.go
For golang/go#42251
Change-Id: I0c49df26a6c695f938c98c9a7b26325d991b3cfb
Reviewed-on: https://go-review.googlesource.com/c/build/+/276032
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change adds the use of TLS certificates when connecting to a
remote buildlet via port 443. This change only applies to the
ConnectSSH, all other remote buildlet connections already use the TLS certificate.
Fixesgolang/go#41697
Change-Id: Ibc3e85edb562c42c9da2b1025f4b291b4a88deaa
Reviewed-on: https://go-review.googlesource.com/c/build/+/258097
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change stops explicitly selecting a random availability zone upon
EC2 vm creation if one is not set. The EC2 api will select an
availability zone with capacity for the request when an availability
zone is not set. The caller maintains the ability to set an
availability zone if needed.
Updates golang/go#36841
Change-Id: I87d4e8e66fcfadfedbe584a2c016696dc4d62062
Reviewed-on: https://go-review.googlesource.com/c/build/+/247901
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change modifies the WaitUntilVMExists method and sets it to
not be exported. It will never be directly called by anything outside
of the EC2Client.
Updates golang/go#36841
Change-Id: If6dfea8c4c191036c8fadd9ad8767e85e55cd224
Reviewed-on: https://go-review.googlesource.com/c/build/+/247400
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Use the cloud package for AWS functions. Remove the unused
destroyVM function.
Updates golang/go#36841
Change-Id: I00e1a20c904f7c4be6460ac302085b28f518d161
Reviewed-on: https://go-review.googlesource.com/c/build/+/236300
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
This enables rundockerbuildlet to run non-reverse buildlet
image on EC2. It will only run a single instance of rundockerbuildlet
once. It exposes port 443 for the coordinator to authenticate with the running
buildlet.
This also adds the buildlet name and buildlet container URL to
the EC2 user data struct retrieved by rundockerbuildlet.
Updates golang/go#36841
Change-Id: I31de754e2ac8970c6f18993104de0e0baea5dc31
Reviewed-on: https://go-review.googlesource.com/c/build/+/234114
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change adds a boolean which identified when an instance is an
EC2 instance. Various EC2 related variables have been renamed to be
more consistent.
Updates golang/go#36841
Change-Id: Ief385bf9d41e320ebd7d31bc0108e8053f337afd
Reviewed-on: https://go-review.googlesource.com/c/build/+/233801
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
This change makes multiple changes to the creation of
EC2 instances via the AWS buildlet:
- Adds a tag resource type.
- Adds an ssh key.
- Instead of waiting for an instance to exist, it now
waits for the instance to be running.
- Renames the user data type.
- It base64 encodes the user data.
Updates golang/go#36841
Change-Id: I90e98dfa11f3a14478580ba4ca4a79724c085be9
Reviewed-on: https://go-review.googlesource.com/c/build/+/233798
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
The buildlet client creation will either timeout at the current
default timeout period or the timeout set in the context. The
existing effective timeout is the five seconds set in the
probeBuildlet function.
Fixesgolang/go#38956
Updates golang/go#36841
Change-Id: I09e96e2c5abcc45ccd535596104c52998ddb0d7a
Reviewed-on: https://go-review.googlesource.com/c/build/+/232997
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
This change adds an AWS buildlet client which allows us to
create EC2 instances on AWS. With this change we have also
moved a portion of the gce creation logic into a helper
function which allows multiple clients to use it. Metadata
for the instances are stored in the user data fields.
The creation of a buildlet pool and modifications to
rundocker buildlet be made in order to enable this change.
Updates golang/go#36841
Change-Id: Ice03e1520513d51a02b9d66542e00012453bf0d9
Reviewed-on: https://go-review.googlesource.com/c/build/+/232077
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
This CL skips deleting active remote buildlets.
The coordinator has multiple ways of tracking stale buildlets. For our
GCE buildlets, we periodically delete old VMs after their expiration
time, typically 45 minutes after their creation. The expiration tracking
in coordinator/gce.go does not account for remote buildlets, which are
buildlets created by users or cmd/release. Remote buildlets have their
own staleness checks and cleanup process, so we should skip the GCE
specific cleanup logic for them.
This adds an additional field to the buildlet Client in order to
correlate a GCE VM with a buildlet.
Updates golang/go#37001
Change-Id: Ib0acdf79c4dfbee6e0061c513f98b749d4b9cc64
Reviewed-on: https://go-review.googlesource.com/c/build/+/217722
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
buildlet should use a specific service account, allowing fine-grained
permission control that is different from the default service account
for whichever project they are run in.
Change-Id: I7a86308d6b65f370dfc49649ef10686d1d8b2974
Reviewed-on: https://go-review.googlesource.com/c/build/+/210958
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
This fixes stuff in CL 210498 and CL 210237.
I renamed the Zone field to ControlZone both to make it more clear and
to force compilation errors wherever Zone was used previously, which
revealed some things that were missed.
Updates golang/go#35987
Change-Id: I2f890727ece86d093a90a3b47701caa58de6ccbc
Reviewed-on: https://go-review.googlesource.com/c/build/+/210541
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
GCE zones can and will run out of resources. This will randomly
select a zone from a list of zones to deploy each new VM to.
Fixesgolang/go#35987
Change-Id: I57acad5c4e81d108f7db8f5bb1ff221a1845a422
Reviewed-on: https://go-review.googlesource.com/c/build/+/210237
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Also modernizes some code in the coordinator.
Updates golang/go#35354 (or fixes. But we could return more info.)
Change-Id: Ifc1aa85ca217a0932e388ec5d36ef0737b90c63d
Reviewed-on: https://go-review.googlesource.com/c/build/+/207841
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Maybe this will solve the golang/go#28365 problems. But at least it
gets us into codepaths that are known & trusted, and removes use of
deprecated API.
Also, add more logging to help debug golang/go#28365.
Updates golang/go#28365
Change-Id: Ibff2b03fd82573cbeedbbc22d12c30ae1a3c3aa0
Reviewed-on: https://go-review.googlesource.com/c/build/+/203217
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Previously, we were invoking a single 'go test' run at the repository
root with the import path pattern of 'golang.org/x/{repo}/...'. This
pattern does not match packages that are located in nested modules
in the repository.
Look for go.mod files in all subdirectories of the repository to find
all inner modules. Then, run 'go test' inside each module root, thus
testing all packages in all modules of the repository. If one of the
test invocations fails, keep testing others, and report all failures.
When looking for inner modules, consider only those that have module
path that would not be ignored by the go tool and aren't vendored.
This way, go.mod files inside testdata directories aren't treated as
if they're proper modules.
This is being done only when the tests are running in module mode,
since module boundaries don't exist in GOPATH mode.
Fixesgolang/go#32528
Change-Id: I9f8558982885c9955d3b34127c80c485d713b380
Reviewed-on: https://go-review.googlesource.com/c/build/+/194559
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Binary releases need to build Go and include binaries such as bin/go,
bin/gofmt, and others. Previously, this was accomplished by running
all.bash script for some GOOS/GOARCH pairs, and make.bash for others
where it wasn't viable to run tests as part of the release process.
This change makes the release process more consistent by always
packaging the release archive file after running make.bash. We still
run all.bash in situations where it was previously run, but we do so
after the release file has already been created. This avoids the
risk of any changes to GOROOT that may occur as part of all.bash
(including changing file permissions to be read-only) being included
in the final release file.
Add a step to check that files in the buildlet's $WORKDIR/go and
$WORKDIR/go/bin directories have expected permissions before
creating the release file.
Fixesgolang/go#33537
Updates golang/go#30316
Change-Id: I7d40716dba656a8aca711377f2995df4880166c5
Reviewed-on: https://go-review.googlesource.com/c/build/+/189537
Reviewed-by: Andrew Bonventre <andybons@golang.org>
The debugnewvm command was printing out the wrong VM name, since the
old method had also assumed (but not documented) that it only applied
to nested virtualization containers. But remove that requirement and
make the empty string mean unspecified instead (which we currently
mean to use Container-Optimized OS) role and then use its new
definition.
Change-Id: Ieca138285aa567b1c24d585c5aa180f8a1534154
Reviewed-on: https://go-review.googlesource.com/c/build/+/177919
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
5 seconds is too slow on linux-arm it seems (starting sshd,
conditionally creating a host key, etc)
Change-Id: I2015908eb892d0b2dd580db8ba96c10726502cbb
Reviewed-on: https://go-review.googlesource.com/c/build/+/175998
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The old revdial has a simple multiplexing protocol that was like
HTTP/2 but without flow control, etc. But it was too simple (no flow
control) and too complex. Instead, just use one TCP connection per
reverse dialed connection. For now, the NAT'ed machine needs to go
re-connect for each incoming connection, but in practice that's just
once.
The old implementation is retained for now until all the buildlets are
updated.
Updates golang/go#31639
Change-Id: Id94c98d2949e695b677531b1221a827573543085
Reviewed-on: https://go-review.googlesource.com/c/build/+/174082
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This adds scripts to create a new builder host image that acts like
Container-Optimized OS (has docker, runs konlet on startup) but with a
Debian 9 kernel + userspace that permits KVM for nested
virtualization.
Updates golang/go#15581 (solaris)
Updates golang/go#23060 (dragonfly)
Updates golang/go#30262 (riscv)
Updates golang/go#30267 (fuchsia)
Updates golang/go#23824 (android)
Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884
Reviewed-on: https://go-review.googlesource.com/c/163301
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This adds a linux-amd64 COS builder that should be just like our
existing linux-amd64 COS builder except that it's using a forked image
that has the VMX license bit enabled for nested virtualization. (GCE
appears to be using the license mechanism as some sort of opt-in
mechanism for features that aren't yet GA; might go away?)
Once this is in, it won't do any new builds as regular+trybot builders
are disabled. But it means I can then use gomote + debugnewvm to work
on preparing the other four image types.
Updates golang/go#15581 (solaris)
Updates golang/go#23060 (dragonfly)
Updates golang/go#30262 (riscv)
Updates golang/go#30267 (fuchsia)
Updates golang/go#23824 (android)
Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015
Reviewed-on: https://go-review.googlesource.com/c/162959
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Once containers run on COS instead of Kubernetes, one name (Kube*) is
wrong and the other (GCE) is ambiguous. So rename them now to be more
specific.
No behavior changes. Just renaming in this step, to reduce size of
next CL.
Updates golang/go#25108
Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b
Reviewed-on: https://go-review.googlesource.com/111639
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This removes some duplication of scopes and how to get the
TokenSource and which credentials to use.
And update the coordinator deps, since its rev of
golang.org/x/oauth2/google was too old to have the new type.
I want to clean this up more, but I need to make some changes to to
the oauth2/google package first. More later.
Change-Id: Ic2799ec2ec62f67c65de6380b373fe915a43003e
Reviewed-on: https://go-review.googlesource.com/111266
Reviewed-by: Andrew Bonventre <andybons@golang.org>
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixesgolang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
My gomote username is josharian.
My whoami is josh.
As a result, I need to run 'gomote -user=josharian'
every time, which is an annoyance.
Add a simple config file to allow a user to change their user name.
On startup, gomote checks whether user-$(whoami).user exists.
If so, it reads it to get the real username.
So I have a config file ~/.config/gomote/user-josh.user
with the contents 'josharian'.
It's a bit of an ugly hack, but it works and spares me some pain.
Change-Id: I372d5a786b99c9e3c6a57f25b0a38b9146f23598
Reviewed-on: https://go-review.googlesource.com/42230
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Replace oauth2.NoContext (deprecated) with context.Background(),
which has been available for two consecutive releases.
Add more cloud.google.com/go packages to the cmd/coordinator
Dockerfile to fix an error building cmd/coordinator. A dependency is
not present in master of cloud.google.com/go, but was present in the
older revision, and was not getting checked out correctly during the
"go get" step. In addition, we were failing to fetch dependencies for
some packages that coordinator depends on. I added instructions for
hopefully doing this more systematically in the future.
Fix the gitmirror Dockerfile which has the same problem.
Change-Id: Id6c2220482350a686b87742ec7915c457a689e52
Reviewed-on: https://go-review.googlesource.com/40852
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Alexander Rakoczy
Heschi Kreinick
Julie Qiu
Carlos Amedee
Brad Fitzpatrick
Dmitri Shuralyov
Josh Bleecher Snyder
Kevin Burke