Граф коммитов

23 Коммитов

Автор SHA1 Сообщение Дата
Brad Fitzpatrick a6dd626c4c all: use "context" instead of golang.org/x/net/context
Change-Id: I106c81a16eb5b39ff38c6f896095a27f597b3f8d
Reviewed-on: https://go-review.googlesource.com/99016
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-03-06 23:26:52 +00:00
Brad Fitzpatrick c5562d003a kubernetes: fix, test GetNodes
Broken in my cleanup golang.org/cl/36692

Change-Id: I5f82314b8ff703cf2c894771fbc0eb16834fc47b
Reviewed-on: https://go-review.googlesource.com/37216
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-17 22:01:21 +00:00
Kevin Burke b7a944e366 all: fix vet errors
It would be nice to run these alongside the tests. There is only one
vet error left in the x/build tree, an unreachable return next to a
TODO.

Change-Id: Ic31d96cdee5bb0f6b9795237ff3b743a4b76e3cf
Reviewed-on: https://go-review.googlesource.com/36898
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-12 17:35:52 +00:00
Brad Fitzpatrick 7449a82969 kubernetes: fix typos
From golang.org/cl/36692

Change-Id: Icc63edd28b687cf30c222a3125371ad912bc6da7
Reviewed-on: https://go-review.googlesource.com/36802
Reviewed-by: Kevin Burke <kev@inburke.com>
Reviewed-by: Matt Layher <mdlayher@gmail.com>
2017-02-11 20:05:12 +00:00
Brad Fitzpatrick b30f5067b7 kubernetes: cleanup, and add DialServicePort
And moves Dial methods to *kubernetes.Client, instead of a separate
Dialer type.

Updates golang/go#18817

Change-Id: I045ac48441b9139cb0e01ceb4969b29cf5e72507
Reviewed-on: https://go-review.googlesource.com/36692
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-02-09 23:20:38 +00:00
Brad Fitzpatrick adc161a243 Add gke package, add kubenetes.Dialer type.
Updates golang/go#18817

Change-Id: Ifee53384486b0692899b77be2eaa42ca9006ef8e
Reviewed-on: https://go-review.googlesource.com/36016
Reviewed-by: Chris Broadfoot <cbro@golang.org>
2017-01-31 21:33:25 +00:00
Brad Fitzpatrick 71265acedb all: adjust things for upgrade from GKE 1.2 to GKE 1.4
We hit GKE bugs and changes when upgrading from GKE 1.2 to 1.4.

The main issue is that Kubernetes does't reserve CPU or memory for
itself on nodes, so things were OOMing and getting killed. And when
Docker or Kubernetes got killed themselves, they were wedging and not
recovering.

So we're going to run a daemonset (POD on all nodes) to reserve space
for Kubernetes for it. That's not in this CL.

But this CL got us limping along and was already in production. It
doubles resource RAM usage for jobs, so fewer things schedule per node.
While we're at it, let jobs use more CPU if it's available.

Also, disable auto-scaling. It was off before by hand. Force it off
programatically too. And make the node count 5, like it was by hand.

Also, force un-graceful pod deletes, since GKE 1.3 or something
introduced a graceful-vs-ungraceful distinction, which we weren't
handling previously and therefore pods never were being deleted.

Change-Id: I3606e4e2e92c496d8194503d510921bd1614d34e
Reviewed-on: https://go-review.googlesource.com/33490
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-12-01 23:48:21 +00:00
Brad Fitzpatrick 304492cf2e kuberentes: roll back recent Kubernetes client changes
https://golang.org/cl/21765 was not tested and not adequately reviewed
and doesn't work at all. Roll it back for now, since the blind fix in
https://golang.org/cl/21765 helped a bit but wasn't enough.

Change-Id: I781fac04d019fa5e6120ec398e13671b89d802c5
Reviewed-on: https://go-review.googlesource.com/25043
Reviewed-by: Johan Euphrosine <proppy@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-07-19 02:49:29 +00:00
Brad Fitzpatrick 49a940a217 kubernetes: fix response status code checking
Fixes bugs from https://golang.org/cl/21765

Change-Id: Ifb68b3a6e8b18d5ce61e8fc01b010cc55f3afa00
Reviewed-on: https://go-review.googlesource.com/25042
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-07-19 02:02:17 +00:00
Johan Euphrosine f46fbc254a build/kubernetes: add secret API methods
- add Get and Create secret
- refactor client
- fix example

Change-Id: Ia5c6f17e86cb01aa8093ec058630b40ee61fa74f
Reviewed-on: https://go-review.googlesource.com/21765
Reviewed-by: Evan Brown <evanbrown@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-07-01 18:31:07 +00:00
Brad Fitzpatrick 4a3f4a8c16 kubernetes: avoid the WatchPod streaming API
Poll every 5 seconds instead.

Change-Id: I0a8a30fee1118b2aecf4b3cbbf342e07afc27602
Reviewed-on: https://go-review.googlesource.com/22820
Reviewed-by: Evan Brown <evanbrown@google.com>
2016-05-05 20:21:37 +00:00
Brad Fitzpatrick c6ced0a944 kubernetes: be more defensive in WatchPod
Change-Id: I0fe9bfb9f0a440a33ff8eedf00940f53b2b7dc87
Reviewed-on: https://go-review.googlesource.com/22789
Reviewed-by: Evan Brown <evanbrown@google.com>
2016-05-04 23:35:15 +00:00
Brad Fitzpatrick 90f7179840 kubernetes: fix potential goroutine leak
Change-Id: I2e9343c7441db43fe7004d877ee0409d29cb9a23
Reviewed-on: https://go-review.googlesource.com/21764
Reviewed-by: Evan Brown <evanbrown@google.com>
2016-04-08 19:09:22 +00:00
Evan Brown 34ff1d9bc8 all: kubernetes builder autoscaling
Improvements to support rapid scheduling of many build jobs:

- Retry logic in Kubernetes client to handle sporadic connection
  closes from their API server under heavy load

- Cluster autoscaler scales on default CPU utilization metric

- Debug mode allows scheduling multiple builds to test scaling

- Account for scheduled vs. provisioned resources in a cluster
  and use that information to estimate when a build's pod
  will be scheduled and in running state

- Use estimated scheduled time to set context timeout

- Track pod lifecycle (requested time, estimated available time,
  actual available time, terminate time, etc)

Change-Id: I14d6c5e01af0970dbb3390a29d1ee5c43049fff8
Reviewed-on: https://go-review.googlesource.com/19524
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-04-07 17:22:57 +00:00
Evan Brown cfb4f34505 kubernetes: improve error message if pod create fails
Updates golang/go#14112

Change-Id: I53e42647e5a8b22cfb70b78fc4571ff78dd23289
Reviewed-on: https://go-review.googlesource.com/18984
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-01-27 22:43:28 +00:00
Brad Fitzpatrick 5f62335c08 cmd/coordinator: fix two crashes
Change-Id: I5ad75b5cd92ba4199642fb2650c7891bbecbed70
Reviewed-on: https://go-review.googlesource.com/17131
Reviewed-by: Evan Brown <evanbrown@google.com>
2015-12-03 18:57:28 +00:00
Evan Brown 2e452e1be8 all: improve Kubernetes API client
* pod creation returns an *api.Pod
* WatchPod replaces WatchPodStatus

Updates golang/go#12546

Change-Id: I34bb6e0d994e552b41a8082cc4672a663ce961a3
Reviewed-on: https://go-review.googlesource.com/17100
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-11-20 17:56:59 +00:00
Evan Brown 83f9748046 all: display pool status and delete failed/old pods
* status page shows kube pool details
* pods created by the coorindator are tracked
* pods that fail to create are deleted
* pods older than delete-at are deleted
* pods created by a different coordinator are deleted

Updates golang/go#12546

Change-Id: I4c4f8ff906962b4a014a66d0a9d490ff17710d62
Reviewed-on: https://go-review.googlesource.com/16101
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-11-05 11:33:24 +00:00
Matthew Dempsky d5f422f474 all: fix vet warnings
Change-Id: Ia4519d9b050136b4d46dc8361f546e3c4205a9cd
Reviewed-on: https://go-review.googlesource.com/16288
Reviewed-by: Dave Cheney <dave@cheney.net>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-10-25 04:14:39 +00:00
Evan Brown 956434c23a all: improved monitoring of buildlet pods after creation
* Replaced cancel with context.Context
* StartPod can be canceled
* Wait for buildlet to come online, but fail fast if pod fails first
* Support timeout waiting for pod to leave pending phase
* Use Kubernetes watch API (long poll)

Updates golang/go#12546

Change-Id: I792a3b8fed615362a0290feee7de0c2cefe43c0e
Reviewed-on: https://go-review.googlesource.com/15285
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-10-15 23:45:50 +00:00
Evan Brown c51d4e0590 all: more work on Kubernetes buildlet pool
Updates golang/go#12546

Change-Id: Iac31c237fed7973ec2bebace5248904fe3e3692d
Reviewed-on: https://go-review.googlesource.com/14396
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-09-08 23:10:58 +00:00
Evan Brown 94367e21d9 kubernetes: update to v1 API
Change-Id: Ia3fe94e944ac80a58eadaa13487a2e14494600d4
Reviewed-on: https://go-review.googlesource.com/14384
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-09-08 18:43:58 +00:00
Johan Euphrosine 093044aa43 kubernetes: add minimal kubernetes API client
Change-Id: I4ca7c27084d5f80f116b8461d0e6769b8bd99ceb
Reviewed-on: https://go-review.googlesource.com/9981
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-06-12 14:50:32 +00:00