Граф коммитов

125 Коммитов

Автор SHA1 Сообщение Дата
Carlos Amedee 489245e0b2 dashboard, env/linux-x86-jessie: remove Debian jessie
Debian jessie is out of LTS support. This change removes
the Debian jessie based builders:
- use linux-x86-stretch for the misc-compile builders.
- linux-x86-jessie host has been removed.
- documentation for the linux-x86-clang builders have been updated.
- documentation throughout the repo has been updated.

For golang/go#47721

Change-Id: Ia2e916ea94fe8e6adb8f5a56d6c7319817de75e2
Reviewed-on: https://go-review.googlesource.com/c/build/+/383154
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Trust: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2022-02-04 16:51:17 +00:00
Carlos Amedee 2adbdc4d98 internal/gomote, buildlet: add the list directory endpoint implementation
This change adds the implementation for the list directory endpoint.
The list directory endpoint lists the contents of a directory on an
gomote instance. The ListDir function on the fake buildlet client has
also been modified to behave more like the actual buildlet client
implementation.

Updates golang/go#48742

Change-Id: If5a1fe9aa62c0dfee56392390a0f4d6d6df7af3d
Reviewed-on: https://go-review.googlesource.com/c/build/+/381738
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-01-31 21:37:25 +00:00
Michael Pratt 7cdbd32ba0 buildlet: migrate back to COS stable milestone
The latest COS version (m93) has released an updated kernel containing
the fix to golang/go#49065, so we no longer need to pin away from it.

For golang/go#49065.

Change-Id: Ie74a432e9229eec5613df7e23ead3f252390ae5f
Reviewed-on: https://go-review.googlesource.com/c/build/+/381454
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
2022-01-28 17:51:30 +00:00
Carlos Amedee 09d18253d4 internal/gomote, internal/gomote/protos: add remove files
This changes the remove directory endpoint to remove files. This change
was made because the endpoint should describe the removal of files or
directories from the gomote instances. It also adds the implementation
of remove files to the gomote server.

Updates golang/go#48742

Change-Id: Icaf11afd760bf9837b8c536703774c9d76e588f6
Reviewed-on: https://go-review.googlesource.com/c/build/+/374155
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-01-14 21:13:55 +00:00
Carlos Amedee 310cfe736f internal/gomote, internal/gomote/protos: add write tgz from URL
This change adds the WriteTGZFromURL implementation to the gomote
server. This RPC allows the caller to instruct the gomote instance to
download the tar.gz file from the provided URL and extract it to the
work directory.

Updates golang/go#48742

Change-Id: I9aefb5dd5fed52dbfc1fba102c38234219e5d56b
Reviewed-on: https://go-review.googlesource.com/c/build/+/377694
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
2022-01-11 23:24:26 +00:00
Carlos Amedee f8a16ea8a8 all: use buildlet client interface to enable testing
This change introduces the use of a buildlet client interface in
existing functions. This is being introduced in order to facilitate
testing.

Change-Id: I41cd5a372fc31edbd9bcba1859cdf84308360174
Reviewed-on: https://go-review.googlesource.com/c/build/+/371818
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
2021-12-16 16:21:21 +00:00
Carlos Amedee 3f88ebde9b internal/coordinator/remote, buildlet: refactor remote buildlet session
This change refactors the existing remote buildlet session management
implementation. It implements logic which was originally found in the
HTTP handlers and moves it into a session manager. This change also
introduces a fake buildlet client intended to be used for testing when
additional functionality is added to the session pool.

Updates golang/go#48735

Change-Id: Ib7d66cefb9847cf2659e7049f707d78536fbbbea
Reviewed-on: https://go-review.googlesource.com/c/build/+/361097
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2021-11-22 20:00:58 +00:00
Heschi Kreinick 3ffd7003c2 buildlet,dashboard: tweaks to Darwin, OpenBSD, and Linux builders
- Add more Monterey builders and remove the known issue: it's stable.
  Also use it for race and nocgo. Update slowbot aliases to point to it.
- Don't test 1.16 on OpenBSD 7.0, it segfaults.
- Pin the Linux (ContainerOS) builders to an older version to avoid the
  pprof kernel bug.

For golang/go#49065, golang/go#49149, and golang/go#48977.

Change-Id: Ibec2fa735183ec65e5066c7c752ac356b7360550
Reviewed-on: https://go-review.googlesource.com/c/build/+/365777
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2021-11-19 20:32:21 +00:00
Heschi Kreinick c17ef9ab22 buildlet: support IAP tunnels for connecting to GCE VMs
Now that we're not allocating external IP addresses to our VMs, you
can't get to them directly from outside our GCP project. The way to fix
that is to use an IAP tunnel. Sadly, there is no API for creating an IAP
tunnel other than gcloud; we have to shell out to it.

Change-Id: I577734aae669a952705276f88f878f040a4b3bf0
Reviewed-on: https://go-review.googlesource.com/c/build/+/364414
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
2021-11-19 18:35:53 +00:00
Alexander Rakoczy 0be17ea117 buildlet: correct MinCPUPlatform name
Also, specify a valid CPU Platform for n2 VMs for OpenBSD.

See: https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform#availablezones

For golang/go#48969

Change-Id: Id503227ad0fbb844e9eaa923b8b3ac7da591ee76
Reviewed-on: https://go-review.googlesource.com/c/build/+/356630
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2021-10-18 20:10:17 +00:00
Alexander Rakoczy 460ff9c2ea buildlet: update MinCPUPlatform for NestedVirt hosts
The n2 machine types that we now use for instances are either Ice Lake
or Cascade Lake. The error message we received when creating a VM was
specifically to use Cascade Lake, not the older Haswell.

For golang/go#48969

Change-Id: I1c05679c2b512ca3bb7dbb0d2efa6f587305f6aa
Reviewed-on: https://go-review.googlesource.com/c/build/+/356629
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
2021-10-18 19:45:58 +00:00
Alexander Rakoczy 922e354eca all: migrate from n1 to e2 instances
We should have similar performance for these machines, and increased
memory availability in some areas.

This is also compliant with an internal policy change.

Change-Id: I07a452a07431322d6fdd4edbb98c5914045d34ed
Reviewed-on: https://go-review.googlesource.com/c/build/+/354757
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
2021-10-13 16:58:04 +00:00
Heschi Kreinick d1c4a25e54 internal/coordinator: start VMs without external IPs
Resubmission of https://golang.org/cl/354642 now that we have our NAT
config sorted. Hopefully.

We're hitting our external IP quota. Now that we have Cloud NAT set up,
we don't need the external IPs at all. Disable them and stop checking
for them.

Change-Id: I5091ebacd79eb0bddec394f35eb9c7368a699ac4
Reviewed-on: https://go-review.googlesource.com/c/build/+/354756
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2021-10-08 21:22:08 +00:00
Heschi Kreinick 2afcc10b04 Revert "internal/coordinator: start VMs without external IPs"
This reverts commit 7e966852b8 (https://golang.org/cl/354642).

Reason for revert: breaks internet access in -longtest builders somehow

Change-Id: I5344e0c8cec5bf93a23c2c3bf592215843e129de
Reviewed-on: https://go-review.googlesource.com/c/build/+/354753
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2021-10-08 19:13:01 +00:00
Heschi Kreinick 7e966852b8 internal/coordinator: start VMs without external IPs
We're hitting our external IP quota. Now that we have Cloud NAT set up,
we don't need the external IPs at all. Disable them and stop checking
for them.

(I don't understand why the code wanted to use an external IP when TLS is
enabled, so I ripped it out. If there's a reason I'm missing let me
know.)

Change-Id: I3e66b2ed984d936aa0f58d3e7178ab7356d73ce1
Reviewed-on: https://go-review.googlesource.com/c/build/+/354642
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2021-10-08 14:23:50 +00:00
Heschi Kreinick 7692cd030d cmd/coordinator: switch to new clusters and network.
For golang/go#48408.

Change-Id: I375daab69d97eefae00e5f9ef3eac88fc2f5e8b8
Reviewed-on: https://go-review.googlesource.com/c/build/+/350790
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2021-09-21 01:12:48 +00:00
Alexander Rakoczy f2096ad995 buildlet: increase timeout period for client
The a1.metal instances take about four minutes to become available
before beginning to boot the Windows ARM64 VM. This means we need
somewhere between five and ten minutes before we can first connect to
the buildlet.

It's possible that other instances are faster to boot, but this is
currently the most economical choice.

For golang/go#42604

Change-Id: Ib76944307c9ad28fd1adca0dc54f3fbeaabfffa0
Reviewed-on: https://go-review.googlesource.com/c/build/+/322654
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
2021-05-26 15:57:38 +00:00
Julie Qiu a56ae9428e all: add or change GoDoc badge to pkg.go.dev with update-readmes.go
update-readmes.go is updated to add a pkg.go.dev badge instead of a
godoc.org badge.

The GoDoc badge is added or changed in all READMEs by running
update-readmes.go

For golang/go#42251

Change-Id: I0c49df26a6c695f938c98c9a7b26325d991b3cfb
Reviewed-on: https://go-review.googlesource.com/c/build/+/276032
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-12-15 04:52:31 +00:00
Carlos Amedee 4e08f9c626 buildlet: use TLS certificates for SSH connections
This change adds the use of TLS certificates when connecting to a
remote buildlet via port 443. This change only applies to the
ConnectSSH, all other remote buildlet connections already use the TLS certificate.

Fixes golang/go#41697

Change-Id: Ibc3e85edb562c42c9da2b1025f4b291b4a88deaa
Reviewed-on: https://go-review.googlesource.com/c/build/+/258097
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-10-07 16:07:34 +00:00
Carlos Amedee 251960799d buildlet: stop explicitly selecting random availability zones on EC2
This change stops explicitly selecting a random availability zone upon
EC2 vm creation if one is not set. The EC2 api will select an
availability zone with capacity for the request when an availability
zone is not set. The caller maintains the ability to set an
availability zone if needed.

Updates golang/go#36841

Change-Id: I87d4e8e66fcfadfedbe584a2c016696dc4d62062
Reviewed-on: https://go-review.googlesource.com/c/build/+/247901
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-08-11 20:47:54 +00:00
Carlos Amedee 5a0d462c32 buildlet: unexport WaitUntilVMExists method
This change modifies the WaitUntilVMExists method and sets it to
not be exported. It will never be directly called by anything outside
of the EC2Client.

Updates golang/go#36841

Change-Id: If6dfea8c4c191036c8fadd9ad8767e85e55cd224
Reviewed-on: https://go-review.googlesource.com/c/build/+/247400
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-08-07 20:48:37 +00:00
Carlos Amedee 87d102082c buildlet: use cloud client with AWS buildlet
Use the cloud package for AWS functions. Remove the unused
destroyVM function.

Updates golang/go#36841

Change-Id: I00e1a20c904f7c4be6460ac302085b28f518d161
Reviewed-on: https://go-review.googlesource.com/c/build/+/236300
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-06-16 13:47:54 +00:00
Carlos Amedee 92f0c56520 cmd/rundockerbuildlet, buildlet: enable running arm EC2 instances
This enables rundockerbuildlet to run non-reverse buildlet
image on EC2. It will only run a single instance of rundockerbuildlet
once. It exposes port 443 for the coordinator to authenticate with the running
buildlet.

This also adds the buildlet name and buildlet container URL to
the EC2 user data struct retrieved by rundockerbuildlet.

Updates golang/go#36841

Change-Id: I31de754e2ac8970c6f18993104de0e0baea5dc31
Reviewed-on: https://go-review.googlesource.com/c/build/+/234114
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-05-15 22:01:25 +00:00
Carlos Amedee 6c8dc32a98 dashboard, buildenv: label EC2 instances
This change adds a boolean which identified when an instance is an
EC2 instance. Various EC2 related variables have been renamed to be
more consistent.

Updates golang/go#36841

Change-Id: Ief385bf9d41e320ebd7d31bc0108e8053f337afd
Reviewed-on: https://go-review.googlesource.com/c/build/+/233801
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-14 02:43:26 +00:00
Carlos Amedee 3b94076fa1 buildlet: modify the request logic and parameters for EC2 instances
This change makes multiple changes to the creation of
EC2 instances via the AWS buildlet:
- Adds a tag resource type.
- Adds an ssh key.
- Instead of waiting for an instance to exist, it now
  waits for the instance to be running.
- Renames the user data type.
- It base64 encodes the user data.

Updates golang/go#36841

Change-Id: I90e98dfa11f3a14478580ba4ca4a79724c085be9
Reviewed-on: https://go-review.googlesource.com/c/build/+/233798
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-13 20:45:43 +00:00
Carlos Amedee bf27e27323 buildlet: ensure buildlet client creation timeout is correct
The buildlet client creation will either timeout at the current
default timeout period or the timeout set in the context. The
existing effective timeout is the five seconds set in the
probeBuildlet function.

Fixes golang/go#38956
Updates golang/go#36841

Change-Id: I09e96e2c5abcc45ccd535596104c52998ddb0d7a
Reviewed-on: https://go-review.googlesource.com/c/build/+/232997
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-08 19:34:32 +00:00
Carlos Amedee 372ecfe489 buildlet: add an AWS buildlet client
This change adds an AWS buildlet client which allows us to
create EC2 instances on AWS. With this change we have also
moved a portion of the gce creation logic into a helper
function which allows multiple clients to use it. Metadata
for the instances are stored in the user data fields.

The creation of a buildlet pool and modifications to
rundocker buildlet be made in order to enable this change.

Updates golang/go#36841

Change-Id: Ice03e1520513d51a02b9d66542e00012453bf0d9
Reviewed-on: https://go-review.googlesource.com/c/build/+/232077
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-05-06 14:00:06 +00:00
Alexander Rakoczy 22bdacadd1 cmd/coordinator,buildlet: keep active GCE SSH sessions alive
This CL skips deleting active remote buildlets.

The coordinator has multiple ways of tracking stale buildlets. For our
GCE buildlets, we periodically delete old VMs after their expiration
time, typically 45 minutes after their creation. The expiration tracking
in coordinator/gce.go does not account for remote buildlets, which are
buildlets created by users or cmd/release. Remote buildlets have their
own staleness checks and cleanup process, so we should skip the GCE
specific cleanup logic for them.

This adds an additional field to the buildlet Client in order to
correlate a GCE VM with a buildlet.

Updates golang/go#37001

Change-Id: Ib0acdf79c4dfbee6e0061c513f98b749d4b9cc64
Reviewed-on: https://go-review.googlesource.com/c/build/+/217722
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-02-06 15:35:01 +00:00
Alexander Rakoczy 65fa539f5d buildlet: use a dedicated service account on GCE
buildlet should use a specific service account, allowing fine-grained
permission control that is different from the default service account
for whichever project they are run in.

Change-Id: I7a86308d6b65f370dfc49649ef10686d1d8b2974
Reviewed-on: https://go-review.googlesource.com/c/build/+/210958
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
2019-12-13 17:33:05 +00:00
Brad Fitzpatrick 326548a346 cmd/coordinator, all: fix more things related to multi-zone buildlets
This fixes stuff in CL 210498 and CL 210237.

I renamed the Zone field to ControlZone both to make it more clear and
to force compilation errors wherever Zone was used previously, which
revealed some things that were missed.

Updates golang/go#35987

Change-Id: I2f890727ece86d093a90a3b47701caa58de6ccbc
Reviewed-on: https://go-review.googlesource.com/c/build/+/210541
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2019-12-09 19:46:22 +00:00
Carlos Amedee 96d53bbd3f cmd/coordinator: support deploying vms in multiple GCE zones
GCE zones can and will run out of resources. This will randomly
select a zone from a list of zones to deploy each new VM to.

Fixes golang/go#35987

Change-Id: I57acad5c4e81d108f7db8f5bb1ff221a1845a422
Reviewed-on: https://go-review.googlesource.com/c/build/+/210237
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-12-06 17:33:13 +00:00
Brad Fitzpatrick 6191a53cff cmd/coordinator, cmd/gomote: show periodic status updates to stderr
Also modernizes some code in the coordinator.

Updates golang/go#35354 (or fixes. But we could return more info.)

Change-Id: Ifc1aa85ca217a0932e388ec5d36ef0737b90c63d
Reviewed-on: https://go-review.googlesource.com/c/build/+/207841
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-20 22:32:37 +00:00
Brad Fitzpatrick d4e4f4e852 buildlet: add context argument to most of the remaining Client methods
(And update all the callers in the tree.)

Updates golang/go#35707

Change-Id: I54769bbe374f31ae1dd07776b27818db91ce8c70
Reviewed-on: https://go-review.googlesource.com/c/build/+/208157
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-11-20 17:29:10 +00:00
Brad Fitzpatrick 5a36865fda buildlet: add context argument to Client.Exec
(And update all the callers in the tree.)

Updates golang/go#35707

Change-Id: I504ef73ea4ba7f8028f47a658c1cd519c7b5d986
Reviewed-on: https://go-review.googlesource.com/c/build/+/207908
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-11-20 06:29:03 +00:00
Brad Fitzpatrick 7c2b75383d cmd/gomote: add gomote rdp subcommand to open an RDP proxy to a Windows buildlet
Fixes golang/go#26090

Change-Id: Ib58fb7767384778b67d0d62d34c1132d70d75c23
Reviewed-on: https://go-review.googlesource.com/c/build/+/207378
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-11-15 18:35:36 +00:00
Brad Fitzpatrick 34578470c8 buildlet, cmd/coordinator: stop using legacy Request.Cancel for cancellation
Maybe this will solve the golang/go#28365 problems. But at least it
gets us into codepaths that are known & trusted, and removes use of
deprecated API.

Also, add more logging to help debug golang/go#28365.

Updates golang/go#28365

Change-Id: Ibff2b03fd82573cbeedbbc22d12c30ae1a3c3aa0
Reviewed-on: https://go-review.googlesource.com/c/build/+/203217
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-24 19:35:26 +00:00
Brad Fitzpatrick f24504d3b6 all: remove use old, overloaded of IN_KUBERNETES
Fixes golang/go#34956

Change-Id: Ie6f521eee5995102d6dc13f92ecca6601a68661b
Reviewed-on: https://go-review.googlesource.com/c/build/+/201739
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-10-17 19:02:58 +00:00
Dmitri Shuralyov 7b90c2e750 cmd/coordinator: find inner modules when testing repos
Previously, we were invoking a single 'go test' run at the repository
root with the import path pattern of 'golang.org/x/{repo}/...'. This
pattern does not match packages that are located in nested modules
in the repository.

Look for go.mod files in all subdirectories of the repository to find
all inner modules. Then, run 'go test' inside each module root, thus
testing all packages in all modules of the repository. If one of the
test invocations fails, keep testing others, and report all failures.

When looking for inner modules, consider only those that have module
path that would not be ignored by the go tool and aren't vendored.
This way, go.mod files inside testdata directories aren't treated as
if they're proper modules.

This is being done only when the tests are running in module mode,
since module boundaries don't exist in GOPATH mode.

Fixes golang/go#32528

Change-Id: I9f8558982885c9955d3b34127c80c485d713b380
Reviewed-on: https://go-review.googlesource.com/c/build/+/194559
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-09-17 16:56:21 +00:00
Dmitri Shuralyov 6225d660dc cmd/release: create release archive after make.bash and before all.bash
Binary releases need to build Go and include binaries such as bin/go,
bin/gofmt, and others. Previously, this was accomplished by running
all.bash script for some GOOS/GOARCH pairs, and make.bash for others
where it wasn't viable to run tests as part of the release process.

This change makes the release process more consistent by always
packaging the release archive file after running make.bash. We still
run all.bash in situations where it was previously run, but we do so
after the release file has already been created. This avoids the
risk of any changes to GOROOT that may occur as part of all.bash
(including changing file permissions to be read-only) being included
in the final release file.

Add a step to check that files in the buildlet's $WORKDIR/go and
$WORKDIR/go/bin directories have expected permissions before
creating the release file.

Fixes golang/go#33537
Updates golang/go#30316

Change-Id: I7d40716dba656a8aca711377f2995df4880166c5
Reviewed-on: https://go-review.googlesource.com/c/build/+/189537
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-08-21 16:40:51 +00:00
Brad Fitzpatrick 6994fc2d58 dashboard, buildlet, cmd/debugnewvm: clean up HostConfig.ContainerVMImage
The debugnewvm command was printing out the wrong VM name, since the
old method had also assumed (but not documented) that it only applied
to nested virtualization containers. But remove that requirement and
make the empty string mean unspecified instead (which we currently
mean to use Container-Optimized OS) role and then use its new
definition.

Change-Id: Ieca138285aa567b1c24d585c5aa180f8a1534154
Reviewed-on: https://go-review.googlesource.com/c/build/+/177919
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-05-21 16:42:08 +00:00
Brad Fitzpatrick 4847158763 buildlet: increase SSH connect timeout
5 seconds is too slow on linux-arm it seems (starting sshd,
conditionally creating a host key, etc)

Change-Id: I2015908eb892d0b2dd580db8ba96c10726502cbb
Reviewed-on: https://go-review.googlesource.com/c/build/+/175998
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-08 16:27:26 +00:00
Brad Fitzpatrick 4f0f4bb614 revdial/v2: add new simpler, non-multiplexing revdial implementation
The old revdial has a simple multiplexing protocol that was like
HTTP/2 but without flow control, etc. But it was too simple (no flow
control) and too complex. Instead, just use one TCP connection per
reverse dialed connection. For now, the NAT'ed machine needs to go
re-connect for each incoming connection, but in practice that's just
once.

The old implementation is retained for now until all the buildlets are
updated.

Updates golang/go#31639

Change-Id: Id94c98d2949e695b677531b1221a827573543085
Reviewed-on: https://go-review.googlesource.com/c/build/+/174082
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-04-29 17:41:24 +00:00
Brad Fitzpatrick 69dd6b2c22 env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx
This adds scripts to create a new builder host image that acts like
Container-Optimized OS (has docker, runs konlet on startup) but with a
Debian 9 kernel + userspace that permits KVM for nested
virtualization.

Updates golang/go#15581 (solaris)
Updates golang/go#23060 (dragonfly)
Updates golang/go#30262 (riscv)
Updates golang/go#30267 (fuchsia)
Updates golang/go#23824 (android)

Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884
Reviewed-on: https://go-review.googlesource.com/c/163301
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-21 22:30:49 +00:00
Brad Fitzpatrick 88cd9dd988 buildlet: change image name for COS-with-vmx buildlet
The COS image I'd forked from earlier didn't have CONFIG_KVM or
CONFIG_KVM_INTEL enabled in its kernel, so even though I'd enabled the
VMX license bit for the VM, the kernel was unable to use it.

Now I've instead rebuilt the ChromiumOS "lakitu" board with a modified
kernel config:

   https://cloud.google.com/container-optimized-os/docs/how-to/building-from-open-source

More docs later. Still tinkering. Nothing uses this yet.

Updates golang/go#15581 (solaris)
Updates golang/go#23060 (dragonfly)
Updates golang/go#30262 (riscv)
Updates golang/go#30267 (fuchsia)
Updates golang/go#23824 (android)

Change-Id: Id2839066e67d9ddda939d96c5f4287af3267a769
Reviewed-on: https://go-review.googlesource.com/c/163057
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-19 20:44:46 +00:00
Brad Fitzpatrick 0261b66eb0 dashboard, buildlet: add a disabled builder with nested virt, for testing
This adds a linux-amd64 COS builder that should be just like our
existing linux-amd64 COS builder except that it's using a forked image
that has the VMX license bit enabled for nested virtualization. (GCE
appears to be using the license mechanism as some sort of opt-in
mechanism for features that aren't yet GA; might go away?)

Once this is in, it won't do any new builds as regular+trybot builders
are disabled. But it means I can then use gomote + debugnewvm to work
on preparing the other four image types.

Updates golang/go#15581 (solaris)
Updates golang/go#23060 (dragonfly)
Updates golang/go#30262 (riscv)
Updates golang/go#30267 (fuchsia)
Updates golang/go#23824 (android)

Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015
Reviewed-on: https://go-review.googlesource.com/c/162959
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-02-15 22:52:44 +00:00
Brad Fitzpatrick e9fe3dc893 buildlet, dashboard: add min CPU platform knob, require Skylake for OpenBSD
I thought this would be enough for OpenBSD to select the TSC on its own without
being forced to (as in CL 160319), but apparently it is not:

    https://github.com/golang/go/issues/29223#issuecomment-459035482

So it seems like we want both this CL and CL 160319.

Updates golang/go#29223

Change-Id: I0a092d62881d8dcce0ef1129d8d32d8f4025b6ac
Reviewed-on: https://go-review.googlesource.com/c/160457
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2019-01-30 21:55:30 +00:00
Brad Fitzpatrick 6ae8750e84 all: use Container-Optimized VMs instead of Kubernetes for buildlet containers
Fixes golang/go#25108

Change-Id: I084669b52b699700ed26a7fdd890d9205a8b9dc9
Reviewed-on: https://go-review.googlesource.com/111267
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-11 03:29:07 +00:00
Brad Fitzpatrick 23fc7515b0 buildlet: remove support for preemptible VMs we haven't used in a long time
Change-Id: I867c4b491327962410768291f8656665c2399e7b
Reviewed-on: https://go-review.googlesource.com/111641
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 17:19:19 +00:00
Brad Fitzpatrick 14d99737ae all: rename Kube to Container and IsGCE to IsVM
Once containers run on COS instead of Kubernetes, one name (Kube*) is
wrong and the other (GCE) is ambiguous. So rename them now to be more
specific.

No behavior changes. Just renaming in this step, to reduce size of
next CL.

Updates golang/go#25108

Change-Id: Ib09eb682ef74acbbf6ed50b46074f834ef5e0c0b
Reviewed-on: https://go-review.googlesource.com/111639
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-05 16:50:07 +00:00
Brad Fitzpatrick 9929a3fe2b all: move to using the oauth2/google.Credentials type, add buildenv accessor
This removes some duplication of scopes and how to get the
TokenSource and which credentials to use.

And update the coordinator deps, since its rev of
golang.org/x/oauth2/google was too old to have the new type.

I want to clean this up more, but I need to make some changes to to
the oauth2/google package first. More later.

Change-Id: Ic2799ec2ec62f67c65de6380b373fe915a43003e
Reviewed-on: https://go-review.googlesource.com/111266
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-05-04 17:04:37 +00:00