Граф коммитов

155 Коммитов

Автор SHA1 Сообщение Дата
Dmitri Shuralyov 1ee051bb32 all: clean up obsolete +build lines and io/ioutil uses [generated]
The module go directive is at 1.20, so modernize it a bit by
applying changes generated by the script below with go1.21.1.

[git-generate]
go fix ./...
gofmt -r 'ioutil.Discard -> io.Discard' -w .
gofmt -r 'ioutil.NopCloser -> io.NopCloser' -w .
gofmt -r 'ioutil.ReadAll -> io.ReadAll' -w .
gofmt -r '"ioutil.ReadAll: %v" -> "io.ReadAll: %v"' -w .
gofmt -r 'ioutil.ReadDir -> os.ReadDir' -w .
gofmt -r 'ioutil.ReadFile -> os.ReadFile' -w .
gofmt -r 'ioutil.TempDir -> os.MkdirTemp' -w .
gofmt -r 'ioutil.TempFile -> os.CreateTemp' -w .
gofmt -r 'ioutil.WriteFile -> os.WriteFile' -w .
goimports -w .

Change-Id: I145b27756e7c6101e992747ee822e7237df4cb91
Reviewed-on: https://go-review.googlesource.com/c/build/+/527016
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2023-09-08 21:45:36 +00:00
Carlos Amedee 0dd4a72118 all: add missing license headers
This adds open source license headers to files that were missing
them. These were discovered by the new linters we are prototyping.

Change-Id: Ie4f5f4a661a0e250422f45b09cadc63fe3386f6f
Reviewed-on: https://go-review.googlesource.com/c/build/+/487395
Reviewed-by: Carlos Amedee <amedee@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Carlos Amedee <amedee@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2023-04-24 13:52:27 +00:00
Michael Pratt 3e1213ad3a buildlet: enable core dump creation on container VMs
Container-optimized OS sets kernel.core_pattern = "|/bin/false",
effectively disabling core dump creation regardless of RLIMIT_CORE.

We have tests that want to analyze core dumps, so reset core_pattern
back to the default value on boot.

For golang/go#25218.

Change-Id: I7e3cc7496a5428326855cc687b87cb4da76fdd66
Reviewed-on: https://go-review.googlesource.com/c/build/+/479557
Run-TryBot: Michael Pratt <mpratt@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2023-03-28 17:48:07 +00:00
cui fliter 459bd2670d all: fix some comments
Change-Id: I15769c4302e8dd1d649e0eaa9b16099f0eda603b
Reviewed-on: https://go-review.googlesource.com/c/build/+/464239
Run-TryBot: Ian Lance Taylor <iant@google.com>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
2023-01-31 14:57:29 +00:00
Carlos Amedee 28f79946a9 all: add additional storage to linux-arm64-longtest builder
This adds the ability to set the size of the root drive for a GCE
host machine. The linux-arm64-longtest boot disk has been increased
from 10GB to 20GB.

Updates golang/go#49649

Change-Id: I74ebf4bd533c91c8716b7df167afacbfdded3ed3
Reviewed-on: https://go-review.googlesource.com/c/build/+/453955
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-11-29 20:06:34 +00:00
Dmitri Shuralyov e73b639045 internal/task: make fakeBuildlet create directories like real buildlet
The new tasks to create a macOS installer uncovered a few places where
the fake buildlet was skipping the step of creating directories like
the real one. Fix that so it behaves more like the real one.

For golang/go#53632.

Change-Id: I5882d916edba92e21e25282a70efed6bd49ad17b
Reviewed-on: https://go-review.googlesource.com/c/build/+/447296
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
2022-11-04 18:53:58 +00:00
Carlos Amedee 2d7d1c6d6f buildlet: use cloud init for ARM64 instances
GCE ARM64 instances do not support the Konnn on COS images. Cloud init
is used to start containers on GCE ARM64 COS images instead of using the
gce-container-declaration metadata method.

Updates golang/go#53851

Change-Id: I3097ab3fc075829a45c93320023e54d3c16fa7a1
Reviewed-on: https://go-review.googlesource.com/c/build/+/447738
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
2022-11-03 20:29:01 +00:00
Carlos Amedee 8f6d74d4ef dashboard, buildlet: add support for ARM64 COS images
This adds the ability to query for COS images which are ARM64 based.
The HostConfig has been updated to provide the ability to note that a
host requires an ARM64 COS image.

Updates golang/go#53851

Change-Id: Ib57bf6359ca365078528dbb6912dca2402939df9
Reviewed-on: https://go-review.googlesource.com/c/build/+/447259
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
2022-11-03 20:28:29 +00:00
Dmitri Shuralyov 21af757eea buildlet: include details in GCEError.Error
Some of our logging in the coordinator includes pointers and repetition,
like so:

2022/09/27 19:16:27 failed with errors: [0xc02b50e120]
2022/09/27 19:16:27 Failed to create VM: "failed with errors: [0xc02b50e120]". Retrying after 1 second (attempt: 2).

The GCE error types only have MarshalJSON methods, so use that to show
more information. (This builds on the previous attempt in CL 428114.)
Remove the duplicate log, since the caller can log or otherwise handle
the error.

For golang/go#48857.

Change-Id: I0d31cac833ea617b4d722751e3ffe9a6689aeb87
Reviewed-on: https://go-review.googlesource.com/c/build/+/435379
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-09-28 15:45:39 +00:00
Alex Rakoczy 1c0f6a04fb buildlet: log something useful for quota errors
As implemented, it just logs a pointer address.

Updates golang/go#48857

Change-Id: Iba5bc8dbbb7a8eb7b5c110cd34233731e4f534ef
Reviewed-on: https://go-review.googlesource.com/c/build/+/428114
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Run-TryBot: Jenny Rakoczy <jenny@golang.org>
2022-09-02 20:06:33 +00:00
Alex Rakoczy edcd807277 buildlet: copy GCE errors
This was intended to be part of CL 426794.

Updates golang/go#48857

Change-Id: Ic317c7fb633910c969f27bd220e17dba148e8909
Reviewed-on: https://go-review.googlesource.com/c/build/+/426798
Run-TryBot: Jenny Rakoczy <jenny@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2022-08-30 21:24:44 +00:00
Alex Rakoczy b8c6553e4d buildlet,internal/coordinator/pool: retry quota exceeded
Handle QUOTA_EXCEEDED errors when trying to create GCE buildlets. Due to
contention with other processes (GKE, etc.) in our builder project, we
may not have a perfect representation of quota usage. Instead, retry as
long as someone is waiting for a builder.

Also, delete some dead code.

Updates golang/go#48857

Change-Id: Iecf48378bdd46a4c817c09436fc45b4e27744ab6
Reviewed-on: https://go-review.googlesource.com/c/build/+/426794
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Auto-Submit: Jenny Rakoczy <jenny@golang.org>
Run-TryBot: Jenny Rakoczy <jenny@golang.org>
2022-08-30 19:39:17 +00:00
Carlos Amedee 5e298dc481 cmd/coordinator: check the session pool for buildlets
Remote buildlets should not be destroyed by the pool implementations for
instances which have been created on clouds. This change adds a check
to ensure that the pool implementations can check for remote buildlets
for instances which are managed by the session pools. Those sessions will
be destroyed by the session pool.

The buildlet client accessors for instance names have been changed so
that they are no longer GCE specific.

The EC2 session pool implementation has been changed so that the
instance name is set in the buildlet client when an EC2 instance is
created.

Updates golang/go#54735
Updates golang/go#54696

Change-Id: Iae3c86cc47b4b79e3287d581dc042dcf2714f7c9
Reviewed-on: https://go-review.googlesource.com/c/build/+/426015
Reviewed-by: Carlos Amedee <carlos@golang.org>
Auto-Submit: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-08-29 16:03:30 +00:00
Heschi Kreinick 8a5c3b95cf cmd/coordinator: remove Kubernetes buildlet support
This was fully disabled for golang/go#25108: PreferContainersOnCOS is true in all
environments. (Note that the comment on that field is wrong AFAICT.)

Notable changes:
- basepin images are created in VMZones[0], which in prod is
  us-central1-a rather than -f.
- I updated the logic for automatically determining the GCE zone/region.
  I don't know when this is used and didn't bother testing it.

Change-Id: I45908383001ccdf21454cdf17abb0863abe822af
Reviewed-on: https://go-review.googlesource.com/c/build/+/422956
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-08-12 16:15:55 +00:00
Heschi Kreinick 082d6eed19 cmd/relui: use gRPC buildlet
Update relui to communicate via gRPC. Also fix a couple silly mistakes
and make our service accounts trusted to use sensitive builders.

Fixes golang/go#54344.

Change-Id: Ie51022d51fd6f17f477534b2d168afef06a30eef
Reviewed-on: https://go-review.googlesource.com/c/build/+/422096
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
2022-08-10 15:11:36 +00:00
Heschi Kreinick f5bac79390 buildlet: create gRPC client
As a first step to port relui over to the gRPC API, create a gRPC client
implementation. This should be suitable for use in the gomote command,
but I don't want to conflict with the giant stack in flight. Also add a
GRPCClient function to iapclient for convenience.

The gomote service was missing working directory information; added it.

I also did some refactoring/cleanup.
- The Client interface is used internally to the coordinator, and as
  such it has a lot of internal functions. The set used by remote
  clients is much more reasonable. Separate it out into a new
  RemoteClient interface.
- AddCloseFn was only used in one place. Delete it.
- DestroyVM was completely unused. Delete it.

For golang/go#54344.

Change-Id: I77344a8c27076e53f565e9af56f8d365ac89b487
Reviewed-on: https://go-review.googlesource.com/c/build/+/422095
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
2022-08-09 20:49:10 +00:00
Russ Cox 43558672db buildlet: try to clean up iap tunnel better
If you run

	./debugnewvm 2>&1 | cat

(or any similar command with a pipe) the cat often doesn't exit,
because a gcloud compute start-iap-tunnel subprocess is holding
up a reference to the pipe even after we kill it. Try to make it shut
down the entire process tree by sending an interrupt signal first.

And guarantee that the problem won't happen even if the processes
aren't cleaned up, by not letting the subprocess share the parent
stdout/stderr in the first place.

Change-Id: I2fcca6717bb34380322e4a816c80c5f0b9270cfa
Reviewed-on: https://go-review.googlesource.com/c/build/+/420805
Run-TryBot: Russ Cox <rsc@golang.org>
Auto-Submit: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2022-08-08 19:11:30 +00:00
Russ Cox 7032644a1b buildlet, debugnewvm: fixes for apparent GCE API changes
GCE is returning a URL like
	https://blahblah/projects/symbolic-datum-552/zones/us-central1-a
in inst.Zone, not just us-central1-a.

This is making the code that expects a zone break, but it also gives us
a way to find the project, which we need in a gcloud invocation in case
the default is not the right one. So update the code to use the new
inst.Zone to its fullest potential.

Also adjust the 'probing buildlet' loop to be just a little less aggressive.
It can take a while before it succeeds, and every failed probe prints a large
Python stack trace, making once per second a bit too much.

Change-Id: I4512aa5498cef34ca43b9c50fc47efdbeca15c60
Reviewed-on: https://go-review.googlesource.com/c/build/+/419076
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-07-26 14:59:03 +00:00
Bryan C. Mills 7b01ecc2da buildlet,cmd/coordinator: clean up and document buildlet path invariants
For golang/go#53957.
Updates golang/go#33598.

Change-Id: Ie2c47dc0dfd82614b70c983f11462b70bd0703d2
Reviewed-on: https://go-review.googlesource.com/c/build/+/367896
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
2022-07-20 16:26:46 +00:00
Carlos Amedee 4de31d16cc dashboard: change the default instance type for GCE builders
This change increases the default resources allocated to builders
running on GCE. Testing has shown a reduction in all.bash build times
for the machines with larger resources. A few unit tests that use the
default sizes have also been updated throughout the repository.

Updates golang/go#17104

Change-Id: I92ba4509bf667da432f011d8f61d2dea7dac5fc4
Reviewed-on: https://go-review.googlesource.com/c/build/+/413420
Reviewed-by: Alex Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2022-06-21 20:38:46 +00:00
Dmitri Shuralyov 33d38b8f07 buildlet: fix Exec to return ErrTimeout on timeout
The coordinator relies on Exec reporting that the given timeout was
exceeded in order to mark a build as failed instead of retrying it.
A refactor resulted in Exec no longer doing that, despite what its
documentation promises, so fix that.

Also add a test since evidence shows that catching a regression can
be helpful.

For golang/go#42699.
Updates golang/go#35707.

Change-Id: Iacef90b83e7b81fad88a33baa6489d5157e3528f
Reviewed-on: https://go-review.googlesource.com/c/build/+/407555
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-05-27 17:17:50 +00:00
Dmitri Shuralyov abbc846f06 buildlet: bump header timeout to 20 seconds
For consistency with "20 seconds" being chosen as an arbitrary timeout
for a network roundtrip and trivial processing in cmd/coordinator in
CL 406215, apply the same change to buildlet.

Change-Id: I889ff8b86d789a6763eb0a32b0ec40050020223e
Reviewed-on: https://go-review.googlesource.com/c/build/+/407554
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
2022-05-21 03:01:34 +00:00
Dmitri Shuralyov 7413a137a5 cmd/coordinator: consolidate and increase global VM deletion timeout
We had a lot of flexibility over timeouts, making their maintenance
harder. Consolidate it to a single timeout in the pool package, and
modify it from 45 minutes to 2 hours.

There's room for improvement in how we maintain this timeout,
but I'm leaving that for future work (with a tracking issue).

Fixes golang/go#52591.
Updates golang/go#52929.
Updates golang/go#49666.
Updates golang/go#42699.

Change-Id: I2ad92648d89a714397bd8b0e1ec490fc9f6d6790
Reviewed-on: https://go-review.googlesource.com/c/build/+/406216
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
2022-05-16 16:01:33 +00:00
Dmitri Shuralyov dd8f4a3179 cmd/coordinator: drop bookkeeping for old revdial v1 buildlets
By now there are no reverse builders that still use revdial v1,
all are version 23 or newer (where revdial v2 support is added).

Most of the code was dropped in 2019 (in CL 208598),
this CL deletes some remaining obsolete bookkeeping.

Updates golang/go#31639.

Change-Id: Icedd48f9342ea88287c5d580f085785c57846a65
Reviewed-on: https://go-review.googlesource.com/c/build/+/406214
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
2022-05-16 15:58:30 +00:00
Heschi Kreinick bbfc067ccf internal/relui: add an integration test for release building
Adds an integration test. The test uses a fake HTTP server to serve the
fake source archive and bootstrap Go. Fake buildlets run locally in temp
directories, but otherwise act mostly like real ones, running real
commands and reading/writing real tar files. Each step produces a dummy output
that can be used for subsequent steps or verified at the end of the
test.

This is more of a smoke test than an exhaustive check of every detail.
Still, it's a lot better than running the workflow against live
buildlets for incremental development.

I had some vague ideas of recording the execution log for each buildlet
into a golden file, but after trying it out I think that it's not worth
it. There really are only 2-3 commands per target. For now I left the
code in case I change my mind.

In the process of writing the test I realized that all the filenames
were wrong, so fix those up.

For golang/go#51797.

Change-Id: Ie3cc8e335047faa8b50bd05ecc0deddaceb55f71
Reviewed-on: https://go-review.googlesource.com/c/build/+/403057
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
2022-05-12 21:55:26 +00:00
Heschi Kreinick b92283589b cmd/release: refactor release build process
cmd/release spends a lot of time manipulating the buildlet's working
directory to build the release tarball/zip/etc. Most of that work is
self-inflicted: we can manipulate archives just fine in code rather than
working remotely to achieve the same effect.

- Build the source archive completely in memory without using a
  buildlet.
- Rather than running around deleting top-level directories, just
  download go/ and fix up the paths in the resulting tar.
- Delete and rename various files in memory, notably including some
  stuff that the releaselet did. The releaselet is now only used to
  build the MSI.
- Rather than stressing about the buildlet not doing parallel gzip, just
  don't do anything fancy when we rebuild the tar. Delete the
  corresponding buildlet code.
- Instead of verifying that permissions are as we expect, just enforce
  them when we build the archives.

For golang/go#51797.

Change-Id: I51c0b8b5575dff1a7ab91b3c0f78bdb0214b4cae
Reviewed-on: https://go-review.googlesource.com/c/build/+/395935
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Alex Rakoczy <alex@golang.org>
2022-04-21 17:37:02 +00:00
Carlos Amedee 2897e139f2 internal/gomote, internal/gomote/protos: add ReadTGZToURL endpoint
This change adds the ReadTGZToURL endpoint implementation. This tars and
zips the directory requested on the gomote instance and uploads it to
GCS. A signed URL is returned which the caller can use to download the file.

For golang/go#47521
Updates golang/go#48742

Change-Id: I5e9574994810b804acb4b9ed9e6bdda68ea26713
Reviewed-on: https://go-review.googlesource.com/c/build/+/397598
Run-TryBot: Carlos Amedee <carlos@golang.org>
Auto-Submit: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
2022-04-12 20:18:53 +00:00
Carlos Amedee 39aa6936b9 internal/gomote, internal/gomote/protos: implement write files from URL
This change adds the implementation for the WriteFileFromURL endpoint.
The caller will be able to add the contents of an HTTP get call to a
file on the gomote instance. They must set the permissions on the
file. Files located on the gomote transfer GCS bucket will be
retrieved using the GCS storage package and authentication vs a
vanilla HTTP call.

For golang/go#47521
Updates golang/go#48742

Change-Id: If9ac24654352433c7a073de08017213223cf9020
Reviewed-on: https://go-review.googlesource.com/c/build/+/397596
Reviewed-by: Heschi Kreinick <heschi@google.com>
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-08 18:21:42 +00:00
Carlos Amedee 289767d327 internal/gomote, buildlet: add execute command implementation
This change adds the implementation for the execute command endpoint.
This enables the caller to execute a command on the gomote instance.
The output from the command will stream back to the caller if requested.

This change also adds helper functions that retrieve the session from
the session pool and the buildlet client.

For golang/go#47521
Updates golang/go#48742

Change-Id: Iec1853667992b5674b07be5e972ac145a6349fca
Reviewed-on: https://go-review.googlesource.com/c/build/+/382494
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
2022-03-21 13:55:17 +00:00
cuishuang 3ab5e7e87a all: fix some typos
Change-Id: I35d764f27ce6a914e3af406b384e49a5183ca639
GitHub-Last-Rev: cec1a722f8
GitHub-Pull-Request: golang/build#44
Reviewed-on: https://go-review.googlesource.com/c/build/+/389597
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-03-18 22:51:25 +00:00
Carlos Amedee 489245e0b2 dashboard, env/linux-x86-jessie: remove Debian jessie
Debian jessie is out of LTS support. This change removes
the Debian jessie based builders:
- use linux-x86-stretch for the misc-compile builders.
- linux-x86-jessie host has been removed.
- documentation for the linux-x86-clang builders have been updated.
- documentation throughout the repo has been updated.

For golang/go#47721

Change-Id: Ia2e916ea94fe8e6adb8f5a56d6c7319817de75e2
Reviewed-on: https://go-review.googlesource.com/c/build/+/383154
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Trust: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2022-02-04 16:51:17 +00:00
Carlos Amedee 2adbdc4d98 internal/gomote, buildlet: add the list directory endpoint implementation
This change adds the implementation for the list directory endpoint.
The list directory endpoint lists the contents of a directory on an
gomote instance. The ListDir function on the fake buildlet client has
also been modified to behave more like the actual buildlet client
implementation.

Updates golang/go#48742

Change-Id: If5a1fe9aa62c0dfee56392390a0f4d6d6df7af3d
Reviewed-on: https://go-review.googlesource.com/c/build/+/381738
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-01-31 21:37:25 +00:00
Michael Pratt 7cdbd32ba0 buildlet: migrate back to COS stable milestone
The latest COS version (m93) has released an updated kernel containing
the fix to golang/go#49065, so we no longer need to pin away from it.

For golang/go#49065.

Change-Id: Ie74a432e9229eec5613df7e23ead3f252390ae5f
Reviewed-on: https://go-review.googlesource.com/c/build/+/381454
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
2022-01-28 17:51:30 +00:00
Carlos Amedee 09d18253d4 internal/gomote, internal/gomote/protos: add remove files
This changes the remove directory endpoint to remove files. This change
was made because the endpoint should describe the removal of files or
directories from the gomote instances. It also adds the implementation
of remove files to the gomote server.

Updates golang/go#48742

Change-Id: Icaf11afd760bf9837b8c536703774c9d76e588f6
Reviewed-on: https://go-review.googlesource.com/c/build/+/374155
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-01-14 21:13:55 +00:00
Carlos Amedee 310cfe736f internal/gomote, internal/gomote/protos: add write tgz from URL
This change adds the WriteTGZFromURL implementation to the gomote
server. This RPC allows the caller to instruct the gomote instance to
download the tar.gz file from the provided URL and extract it to the
work directory.

Updates golang/go#48742

Change-Id: I9aefb5dd5fed52dbfc1fba102c38234219e5d56b
Reviewed-on: https://go-review.googlesource.com/c/build/+/377694
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
2022-01-11 23:24:26 +00:00
Carlos Amedee f8a16ea8a8 all: use buildlet client interface to enable testing
This change introduces the use of a buildlet client interface in
existing functions. This is being introduced in order to facilitate
testing.

Change-Id: I41cd5a372fc31edbd9bcba1859cdf84308360174
Reviewed-on: https://go-review.googlesource.com/c/build/+/371818
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
2021-12-16 16:21:21 +00:00
Carlos Amedee 3f88ebde9b internal/coordinator/remote, buildlet: refactor remote buildlet session
This change refactors the existing remote buildlet session management
implementation. It implements logic which was originally found in the
HTTP handlers and moves it into a session manager. This change also
introduces a fake buildlet client intended to be used for testing when
additional functionality is added to the session pool.

Updates golang/go#48735

Change-Id: Ib7d66cefb9847cf2659e7049f707d78536fbbbea
Reviewed-on: https://go-review.googlesource.com/c/build/+/361097
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2021-11-22 20:00:58 +00:00
Heschi Kreinick 3ffd7003c2 buildlet,dashboard: tweaks to Darwin, OpenBSD, and Linux builders
- Add more Monterey builders and remove the known issue: it's stable.
  Also use it for race and nocgo. Update slowbot aliases to point to it.
- Don't test 1.16 on OpenBSD 7.0, it segfaults.
- Pin the Linux (ContainerOS) builders to an older version to avoid the
  pprof kernel bug.

For golang/go#49065, golang/go#49149, and golang/go#48977.

Change-Id: Ibec2fa735183ec65e5066c7c752ac356b7360550
Reviewed-on: https://go-review.googlesource.com/c/build/+/365777
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2021-11-19 20:32:21 +00:00
Heschi Kreinick c17ef9ab22 buildlet: support IAP tunnels for connecting to GCE VMs
Now that we're not allocating external IP addresses to our VMs, you
can't get to them directly from outside our GCP project. The way to fix
that is to use an IAP tunnel. Sadly, there is no API for creating an IAP
tunnel other than gcloud; we have to shell out to it.

Change-Id: I577734aae669a952705276f88f878f040a4b3bf0
Reviewed-on: https://go-review.googlesource.com/c/build/+/364414
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
2021-11-19 18:35:53 +00:00
Alexander Rakoczy 0be17ea117 buildlet: correct MinCPUPlatform name
Also, specify a valid CPU Platform for n2 VMs for OpenBSD.

See: https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform#availablezones

For golang/go#48969

Change-Id: Id503227ad0fbb844e9eaa923b8b3ac7da591ee76
Reviewed-on: https://go-review.googlesource.com/c/build/+/356630
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2021-10-18 20:10:17 +00:00
Alexander Rakoczy 460ff9c2ea buildlet: update MinCPUPlatform for NestedVirt hosts
The n2 machine types that we now use for instances are either Ice Lake
or Cascade Lake. The error message we received when creating a VM was
specifically to use Cascade Lake, not the older Haswell.

For golang/go#48969

Change-Id: I1c05679c2b512ca3bb7dbb0d2efa6f587305f6aa
Reviewed-on: https://go-review.googlesource.com/c/build/+/356629
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
2021-10-18 19:45:58 +00:00
Alexander Rakoczy 922e354eca all: migrate from n1 to e2 instances
We should have similar performance for these machines, and increased
memory availability in some areas.

This is also compliant with an internal policy change.

Change-Id: I07a452a07431322d6fdd4edbb98c5914045d34ed
Reviewed-on: https://go-review.googlesource.com/c/build/+/354757
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
2021-10-13 16:58:04 +00:00
Heschi Kreinick d1c4a25e54 internal/coordinator: start VMs without external IPs
Resubmission of https://golang.org/cl/354642 now that we have our NAT
config sorted. Hopefully.

We're hitting our external IP quota. Now that we have Cloud NAT set up,
we don't need the external IPs at all. Disable them and stop checking
for them.

Change-Id: I5091ebacd79eb0bddec394f35eb9c7368a699ac4
Reviewed-on: https://go-review.googlesource.com/c/build/+/354756
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2021-10-08 21:22:08 +00:00
Heschi Kreinick 2afcc10b04 Revert "internal/coordinator: start VMs without external IPs"
This reverts commit 7e966852b8 (https://golang.org/cl/354642).

Reason for revert: breaks internet access in -longtest builders somehow

Change-Id: I5344e0c8cec5bf93a23c2c3bf592215843e129de
Reviewed-on: https://go-review.googlesource.com/c/build/+/354753
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2021-10-08 19:13:01 +00:00
Heschi Kreinick 7e966852b8 internal/coordinator: start VMs without external IPs
We're hitting our external IP quota. Now that we have Cloud NAT set up,
we don't need the external IPs at all. Disable them and stop checking
for them.

(I don't understand why the code wanted to use an external IP when TLS is
enabled, so I ripped it out. If there's a reason I'm missing let me
know.)

Change-Id: I3e66b2ed984d936aa0f58d3e7178ab7356d73ce1
Reviewed-on: https://go-review.googlesource.com/c/build/+/354642
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2021-10-08 14:23:50 +00:00
Heschi Kreinick 7692cd030d cmd/coordinator: switch to new clusters and network.
For golang/go#48408.

Change-Id: I375daab69d97eefae00e5f9ef3eac88fc2f5e8b8
Reviewed-on: https://go-review.googlesource.com/c/build/+/350790
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2021-09-21 01:12:48 +00:00
Alexander Rakoczy f2096ad995 buildlet: increase timeout period for client
The a1.metal instances take about four minutes to become available
before beginning to boot the Windows ARM64 VM. This means we need
somewhere between five and ten minutes before we can first connect to
the buildlet.

It's possible that other instances are faster to boot, but this is
currently the most economical choice.

For golang/go#42604

Change-Id: Ib76944307c9ad28fd1adca0dc54f3fbeaabfffa0
Reviewed-on: https://go-review.googlesource.com/c/build/+/322654
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
2021-05-26 15:57:38 +00:00
Julie Qiu a56ae9428e all: add or change GoDoc badge to pkg.go.dev with update-readmes.go
update-readmes.go is updated to add a pkg.go.dev badge instead of a
godoc.org badge.

The GoDoc badge is added or changed in all READMEs by running
update-readmes.go

For golang/go#42251

Change-Id: I0c49df26a6c695f938c98c9a7b26325d991b3cfb
Reviewed-on: https://go-review.googlesource.com/c/build/+/276032
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-12-15 04:52:31 +00:00
Carlos Amedee 4e08f9c626 buildlet: use TLS certificates for SSH connections
This change adds the use of TLS certificates when connecting to a
remote buildlet via port 443. This change only applies to the
ConnectSSH, all other remote buildlet connections already use the TLS certificate.

Fixes golang/go#41697

Change-Id: Ibc3e85edb562c42c9da2b1025f4b291b4a88deaa
Reviewed-on: https://go-review.googlesource.com/c/build/+/258097
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-10-07 16:07:34 +00:00
Carlos Amedee 251960799d buildlet: stop explicitly selecting random availability zones on EC2
This change stops explicitly selecting a random availability zone upon
EC2 vm creation if one is not set. The EC2 api will select an
availability zone with capacity for the request when an availability
zone is not set. The caller maintains the ability to set an
availability zone if needed.

Updates golang/go#36841

Change-Id: I87d4e8e66fcfadfedbe584a2c016696dc4d62062
Reviewed-on: https://go-review.googlesource.com/c/build/+/247901
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-11 20:47:54 +00:00