Debian jessie is out of LTS support. This change removes
the Debian jessie based builders:
- use linux-x86-stretch for the misc-compile builders.
- linux-x86-jessie host has been removed.
- documentation for the linux-x86-clang builders have been updated.
- documentation throughout the repo has been updated.
For golang/go#47721
Change-Id: Ia2e916ea94fe8e6adb8f5a56d6c7319817de75e2
Reviewed-on: https://go-review.googlesource.com/c/build/+/383154
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Trust: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change adds the implementation for the list directory endpoint.
The list directory endpoint lists the contents of a directory on an
gomote instance. The ListDir function on the fake buildlet client has
also been modified to behave more like the actual buildlet client
implementation.
Updates golang/go#48742
Change-Id: If5a1fe9aa62c0dfee56392390a0f4d6d6df7af3d
Reviewed-on: https://go-review.googlesource.com/c/build/+/381738
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
The latest COS version (m93) has released an updated kernel containing
the fix to golang/go#49065, so we no longer need to pin away from it.
For golang/go#49065.
Change-Id: Ie74a432e9229eec5613df7e23ead3f252390ae5f
Reviewed-on: https://go-review.googlesource.com/c/build/+/381454
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
This changes the remove directory endpoint to remove files. This change
was made because the endpoint should describe the removal of files or
directories from the gomote instances. It also adds the implementation
of remove files to the gomote server.
Updates golang/go#48742
Change-Id: Icaf11afd760bf9837b8c536703774c9d76e588f6
Reviewed-on: https://go-review.googlesource.com/c/build/+/374155
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
This change adds the WriteTGZFromURL implementation to the gomote
server. This RPC allows the caller to instruct the gomote instance to
download the tar.gz file from the provided URL and extract it to the
work directory.
Updates golang/go#48742
Change-Id: I9aefb5dd5fed52dbfc1fba102c38234219e5d56b
Reviewed-on: https://go-review.googlesource.com/c/build/+/377694
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
This change introduces the use of a buildlet client interface in
existing functions. This is being introduced in order to facilitate
testing.
Change-Id: I41cd5a372fc31edbd9bcba1859cdf84308360174
Reviewed-on: https://go-review.googlesource.com/c/build/+/371818
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
This change refactors the existing remote buildlet session management
implementation. It implements logic which was originally found in the
HTTP handlers and moves it into a session manager. This change also
introduces a fake buildlet client intended to be used for testing when
additional functionality is added to the session pool.
Updates golang/go#48735
Change-Id: Ib7d66cefb9847cf2659e7049f707d78536fbbbea
Reviewed-on: https://go-review.googlesource.com/c/build/+/361097
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
- Add more Monterey builders and remove the known issue: it's stable.
Also use it for race and nocgo. Update slowbot aliases to point to it.
- Don't test 1.16 on OpenBSD 7.0, it segfaults.
- Pin the Linux (ContainerOS) builders to an older version to avoid the
pprof kernel bug.
For golang/go#49065, golang/go#49149, and golang/go#48977.
Change-Id: Ibec2fa735183ec65e5066c7c752ac356b7360550
Reviewed-on: https://go-review.googlesource.com/c/build/+/365777
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Now that we're not allocating external IP addresses to our VMs, you
can't get to them directly from outside our GCP project. The way to fix
that is to use an IAP tunnel. Sadly, there is no API for creating an IAP
tunnel other than gcloud; we have to shell out to it.
Change-Id: I577734aae669a952705276f88f878f040a4b3bf0
Reviewed-on: https://go-review.googlesource.com/c/build/+/364414
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
The n2 machine types that we now use for instances are either Ice Lake
or Cascade Lake. The error message we received when creating a VM was
specifically to use Cascade Lake, not the older Haswell.
For golang/go#48969
Change-Id: I1c05679c2b512ca3bb7dbb0d2efa6f587305f6aa
Reviewed-on: https://go-review.googlesource.com/c/build/+/356629
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
We should have similar performance for these machines, and increased
memory availability in some areas.
This is also compliant with an internal policy change.
Change-Id: I07a452a07431322d6fdd4edbb98c5914045d34ed
Reviewed-on: https://go-review.googlesource.com/c/build/+/354757
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Resubmission of https://golang.org/cl/354642 now that we have our NAT
config sorted. Hopefully.
We're hitting our external IP quota. Now that we have Cloud NAT set up,
we don't need the external IPs at all. Disable them and stop checking
for them.
Change-Id: I5091ebacd79eb0bddec394f35eb9c7368a699ac4
Reviewed-on: https://go-review.googlesource.com/c/build/+/354756
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
This reverts commit 7e966852b8 (https://golang.org/cl/354642).
Reason for revert: breaks internet access in -longtest builders somehow
Change-Id: I5344e0c8cec5bf93a23c2c3bf592215843e129de
Reviewed-on: https://go-review.googlesource.com/c/build/+/354753
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
We're hitting our external IP quota. Now that we have Cloud NAT set up,
we don't need the external IPs at all. Disable them and stop checking
for them.
(I don't understand why the code wanted to use an external IP when TLS is
enabled, so I ripped it out. If there's a reason I'm missing let me
know.)
Change-Id: I3e66b2ed984d936aa0f58d3e7178ab7356d73ce1
Reviewed-on: https://go-review.googlesource.com/c/build/+/354642
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
The a1.metal instances take about four minutes to become available
before beginning to boot the Windows ARM64 VM. This means we need
somewhere between five and ten minutes before we can first connect to
the buildlet.
It's possible that other instances are faster to boot, but this is
currently the most economical choice.
For golang/go#42604
Change-Id: Ib76944307c9ad28fd1adca0dc54f3fbeaabfffa0
Reviewed-on: https://go-review.googlesource.com/c/build/+/322654
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
update-readmes.go is updated to add a pkg.go.dev badge instead of a
godoc.org badge.
The GoDoc badge is added or changed in all READMEs by running
update-readmes.go
For golang/go#42251
Change-Id: I0c49df26a6c695f938c98c9a7b26325d991b3cfb
Reviewed-on: https://go-review.googlesource.com/c/build/+/276032
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change adds the use of TLS certificates when connecting to a
remote buildlet via port 443. This change only applies to the
ConnectSSH, all other remote buildlet connections already use the TLS certificate.
Fixesgolang/go#41697
Change-Id: Ibc3e85edb562c42c9da2b1025f4b291b4a88deaa
Reviewed-on: https://go-review.googlesource.com/c/build/+/258097
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change stops explicitly selecting a random availability zone upon
EC2 vm creation if one is not set. The EC2 api will select an
availability zone with capacity for the request when an availability
zone is not set. The caller maintains the ability to set an
availability zone if needed.
Updates golang/go#36841
Change-Id: I87d4e8e66fcfadfedbe584a2c016696dc4d62062
Reviewed-on: https://go-review.googlesource.com/c/build/+/247901
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change modifies the WaitUntilVMExists method and sets it to
not be exported. It will never be directly called by anything outside
of the EC2Client.
Updates golang/go#36841
Change-Id: If6dfea8c4c191036c8fadd9ad8767e85e55cd224
Reviewed-on: https://go-review.googlesource.com/c/build/+/247400
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Use the cloud package for AWS functions. Remove the unused
destroyVM function.
Updates golang/go#36841
Change-Id: I00e1a20c904f7c4be6460ac302085b28f518d161
Reviewed-on: https://go-review.googlesource.com/c/build/+/236300
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
This enables rundockerbuildlet to run non-reverse buildlet
image on EC2. It will only run a single instance of rundockerbuildlet
once. It exposes port 443 for the coordinator to authenticate with the running
buildlet.
This also adds the buildlet name and buildlet container URL to
the EC2 user data struct retrieved by rundockerbuildlet.
Updates golang/go#36841
Change-Id: I31de754e2ac8970c6f18993104de0e0baea5dc31
Reviewed-on: https://go-review.googlesource.com/c/build/+/234114
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change adds a boolean which identified when an instance is an
EC2 instance. Various EC2 related variables have been renamed to be
more consistent.
Updates golang/go#36841
Change-Id: Ief385bf9d41e320ebd7d31bc0108e8053f337afd
Reviewed-on: https://go-review.googlesource.com/c/build/+/233801
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
This change makes multiple changes to the creation of
EC2 instances via the AWS buildlet:
- Adds a tag resource type.
- Adds an ssh key.
- Instead of waiting for an instance to exist, it now
waits for the instance to be running.
- Renames the user data type.
- It base64 encodes the user data.
Updates golang/go#36841
Change-Id: I90e98dfa11f3a14478580ba4ca4a79724c085be9
Reviewed-on: https://go-review.googlesource.com/c/build/+/233798
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
The buildlet client creation will either timeout at the current
default timeout period or the timeout set in the context. The
existing effective timeout is the five seconds set in the
probeBuildlet function.
Fixesgolang/go#38956
Updates golang/go#36841
Change-Id: I09e96e2c5abcc45ccd535596104c52998ddb0d7a
Reviewed-on: https://go-review.googlesource.com/c/build/+/232997
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
This change adds an AWS buildlet client which allows us to
create EC2 instances on AWS. With this change we have also
moved a portion of the gce creation logic into a helper
function which allows multiple clients to use it. Metadata
for the instances are stored in the user data fields.
The creation of a buildlet pool and modifications to
rundocker buildlet be made in order to enable this change.
Updates golang/go#36841
Change-Id: Ice03e1520513d51a02b9d66542e00012453bf0d9
Reviewed-on: https://go-review.googlesource.com/c/build/+/232077
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
This CL skips deleting active remote buildlets.
The coordinator has multiple ways of tracking stale buildlets. For our
GCE buildlets, we periodically delete old VMs after their expiration
time, typically 45 minutes after their creation. The expiration tracking
in coordinator/gce.go does not account for remote buildlets, which are
buildlets created by users or cmd/release. Remote buildlets have their
own staleness checks and cleanup process, so we should skip the GCE
specific cleanup logic for them.
This adds an additional field to the buildlet Client in order to
correlate a GCE VM with a buildlet.
Updates golang/go#37001
Change-Id: Ib0acdf79c4dfbee6e0061c513f98b749d4b9cc64
Reviewed-on: https://go-review.googlesource.com/c/build/+/217722
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
buildlet should use a specific service account, allowing fine-grained
permission control that is different from the default service account
for whichever project they are run in.
Change-Id: I7a86308d6b65f370dfc49649ef10686d1d8b2974
Reviewed-on: https://go-review.googlesource.com/c/build/+/210958
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
This fixes stuff in CL 210498 and CL 210237.
I renamed the Zone field to ControlZone both to make it more clear and
to force compilation errors wherever Zone was used previously, which
revealed some things that were missed.
Updates golang/go#35987
Change-Id: I2f890727ece86d093a90a3b47701caa58de6ccbc
Reviewed-on: https://go-review.googlesource.com/c/build/+/210541
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
GCE zones can and will run out of resources. This will randomly
select a zone from a list of zones to deploy each new VM to.
Fixesgolang/go#35987
Change-Id: I57acad5c4e81d108f7db8f5bb1ff221a1845a422
Reviewed-on: https://go-review.googlesource.com/c/build/+/210237
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Also modernizes some code in the coordinator.
Updates golang/go#35354 (or fixes. But we could return more info.)
Change-Id: Ifc1aa85ca217a0932e388ec5d36ef0737b90c63d
Reviewed-on: https://go-review.googlesource.com/c/build/+/207841
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Maybe this will solve the golang/go#28365 problems. But at least it
gets us into codepaths that are known & trusted, and removes use of
deprecated API.
Also, add more logging to help debug golang/go#28365.
Updates golang/go#28365
Change-Id: Ibff2b03fd82573cbeedbbc22d12c30ae1a3c3aa0
Reviewed-on: https://go-review.googlesource.com/c/build/+/203217
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Previously, we were invoking a single 'go test' run at the repository
root with the import path pattern of 'golang.org/x/{repo}/...'. This
pattern does not match packages that are located in nested modules
in the repository.
Look for go.mod files in all subdirectories of the repository to find
all inner modules. Then, run 'go test' inside each module root, thus
testing all packages in all modules of the repository. If one of the
test invocations fails, keep testing others, and report all failures.
When looking for inner modules, consider only those that have module
path that would not be ignored by the go tool and aren't vendored.
This way, go.mod files inside testdata directories aren't treated as
if they're proper modules.
This is being done only when the tests are running in module mode,
since module boundaries don't exist in GOPATH mode.
Fixesgolang/go#32528
Change-Id: I9f8558982885c9955d3b34127c80c485d713b380
Reviewed-on: https://go-review.googlesource.com/c/build/+/194559
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Binary releases need to build Go and include binaries such as bin/go,
bin/gofmt, and others. Previously, this was accomplished by running
all.bash script for some GOOS/GOARCH pairs, and make.bash for others
where it wasn't viable to run tests as part of the release process.
This change makes the release process more consistent by always
packaging the release archive file after running make.bash. We still
run all.bash in situations where it was previously run, but we do so
after the release file has already been created. This avoids the
risk of any changes to GOROOT that may occur as part of all.bash
(including changing file permissions to be read-only) being included
in the final release file.
Add a step to check that files in the buildlet's $WORKDIR/go and
$WORKDIR/go/bin directories have expected permissions before
creating the release file.
Fixesgolang/go#33537
Updates golang/go#30316
Change-Id: I7d40716dba656a8aca711377f2995df4880166c5
Reviewed-on: https://go-review.googlesource.com/c/build/+/189537
Reviewed-by: Andrew Bonventre <andybons@golang.org>
The debugnewvm command was printing out the wrong VM name, since the
old method had also assumed (but not documented) that it only applied
to nested virtualization containers. But remove that requirement and
make the empty string mean unspecified instead (which we currently
mean to use Container-Optimized OS) role and then use its new
definition.
Change-Id: Ieca138285aa567b1c24d585c5aa180f8a1534154
Reviewed-on: https://go-review.googlesource.com/c/build/+/177919
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
5 seconds is too slow on linux-arm it seems (starting sshd,
conditionally creating a host key, etc)
Change-Id: I2015908eb892d0b2dd580db8ba96c10726502cbb
Reviewed-on: https://go-review.googlesource.com/c/build/+/175998
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The old revdial has a simple multiplexing protocol that was like
HTTP/2 but without flow control, etc. But it was too simple (no flow
control) and too complex. Instead, just use one TCP connection per
reverse dialed connection. For now, the NAT'ed machine needs to go
re-connect for each incoming connection, but in practice that's just
once.
The old implementation is retained for now until all the buildlets are
updated.
Updates golang/go#31639
Change-Id: Id94c98d2949e695b677531b1221a827573543085
Reviewed-on: https://go-review.googlesource.com/c/build/+/174082
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This adds scripts to create a new builder host image that acts like
Container-Optimized OS (has docker, runs konlet on startup) but with a
Debian 9 kernel + userspace that permits KVM for nested
virtualization.
Updates golang/go#15581 (solaris)
Updates golang/go#23060 (dragonfly)
Updates golang/go#30262 (riscv)
Updates golang/go#30267 (fuchsia)
Updates golang/go#23824 (android)
Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884
Reviewed-on: https://go-review.googlesource.com/c/163301
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This adds a linux-amd64 COS builder that should be just like our
existing linux-amd64 COS builder except that it's using a forked image
that has the VMX license bit enabled for nested virtualization. (GCE
appears to be using the license mechanism as some sort of opt-in
mechanism for features that aren't yet GA; might go away?)
Once this is in, it won't do any new builds as regular+trybot builders
are disabled. But it means I can then use gomote + debugnewvm to work
on preparing the other four image types.
Updates golang/go#15581 (solaris)
Updates golang/go#23060 (dragonfly)
Updates golang/go#30262 (riscv)
Updates golang/go#30267 (fuchsia)
Updates golang/go#23824 (android)
Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015
Reviewed-on: https://go-review.googlesource.com/c/162959
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Once containers run on COS instead of Kubernetes, one name (Kube*) is
wrong and the other (GCE) is ambiguous. So rename them now to be more
specific.
No behavior changes. Just renaming in this step, to reduce size of
next CL.
Updates golang/go#25108
Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b
Reviewed-on: https://go-review.googlesource.com/111639
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This removes some duplication of scopes and how to get the
TokenSource and which credentials to use.
And update the coordinator deps, since its rev of
golang.org/x/oauth2/google was too old to have the new type.
I want to clean this up more, but I need to make some changes to to
the oauth2/google package first. More later.
Change-Id: Ic2799ec2ec62f67c65de6380b373fe915a43003e
Reviewed-on: https://go-review.googlesource.com/111266
Reviewed-by: Andrew Bonventre <andybons@golang.org>