The one remaining error reported by "go vet" is the un-canceled
context in cmd/coordinator/remote.go, which isn't an error.
Change-Id: Ie8d9269fc77203cf82d0a724dd33e4871a8113a7
Reviewed-on: https://go-review.googlesource.com/39350
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Capitalized a value and forgot to update the tests.
Change-Id: Idd5bd6fddc4e013d64cdd428dfcbfae137e945a1
Reviewed-on: https://go-review.googlesource.com/39034
Reviewed-by: Kevin Burke <kev@inburke.com>
When we add a meta commit to the corpus, find the CL's status (buried
in the meta commit message or one of its parent commits) and store it
on the CL object.
Add ForeachOpenCL to allow callers to take an action for each open
Gerrit CL. This requires exporting the existing Gerrit and Git
structs.
Change-Id: Ic72721e437c0281244c2d6fb2eb5435f06a7acb5
Reviewed-on: https://go-review.googlesource.com/38724
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Saves a tiny amount of memory, but many fewer allocation.
name old time/op new time/op delta
Get-4 4.97s ±19% 5.22s ± 9% ~ (p=0.421 n=5+5)
name old alloc/op new alloc/op delta
Get-4 1.10GB ± 0% 1.09GB ± 0% -0.67% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
Get-4 14.2M ± 0% 12.9M ± 0% -9.38% (p=0.008 n=5+5)
Change-Id: Ia4d383b7115a856de517002357c069e6b3eee7ef
Reviewed-on: https://go-review.googlesource.com/38782
Reviewed-by: Kevin Burke <kev@inburke.com>
regexp was 25% of the CPU. Now it's lost in the long tail.
Change-Id: Ifc522d0690dff2e2524ec12354d3790946e5a54a
Reviewed-on: https://go-review.googlesource.com/38725
Reviewed-by: Kevin Burke <kev@inburke.com>
The log file gets 50% larger, but loads in half the time.
Change-Id: I0b163f7b39269a5b39bbd51a80a4982682e94863
Reviewed-on: https://go-review.googlesource.com/38723
Reviewed-by: Kevin Burke <kev@inburke.com>
Many revisions of git commits in Gerrit have the same commit message.
Share that memory.
Saves 11 MB (387 MB -> 376 MB).
Change-Id: I39f05cf309404f0b1d55c296a2fe551eb501ae9c
Reviewed-on: https://go-review.googlesource.com/38718
Reviewed-by: Kevin Burke <kev@inburke.com>
Don't update Gerrit refs until after corresponding git commit
mutations are added and processed. Do the batching earlier, so we
always make forward progress.
Also better logging so you can see what's happening.
Change-Id: I493071af7bccec5abf156025389376b9fc182dc8
Reviewed-on: https://go-review.googlesource.com/38672
Reviewed-by: Kevin Burke <kev@inburke.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
StartLogging seemed unnecessary. And processMutation was special in
that it added to the log. The other dozen or so methods named
processMutation or similar were just about process mutations already
in the log. Disambiguate.
Change-Id: I4b4bf46c3ae7996ad5f7e06de5456f4c06b6bb40
Reviewed-on: https://go-review.googlesource.com/38658
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Pass --watch-gerrit on the command line to watch a Gerrit project. We
initialize a Git repository in $HOME/var/maintnerd/<gerrit-url>, then
fetch all metadata related to that project, and store it in the
Corpus.
Change-Id: I47a8b5e5af2ae1b99c97cc756790c8e21465e8ee
Reviewed-on: https://go-review.googlesource.com/38421
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The bot has the current functionality:
* auto-lock old issues and add the "FrozenDueToAge" label.
(this CL deletes and moves the old "issuelock" code into here)
* auto-label and auto-milestone proposals (bugs with title prefix
"Proposal:")
* auto-milestones "x/pkg: " bugs to "Unreleased" milestone unless
the pkg is in a whitelist of packages known to be vendored into
std.
* auto-milestones "gccgo" bugs to milestone "Gccgo"
* auto-labels builder bugs with label "Builders"
* auto-labels documentation bugs with label "Documentation"
* auto-closes bugs sitting with "WaitingForInfo" label state for
a month if the original author hasn't replied
Change-Id: I6cb7bfd3ade5aab62ac4973b79cc733c9b1fbfb3
Reviewed-on: https://go-review.googlesource.com/38385
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This also adds a maintnerd flag --sync-and-quit.
And unexports some internal guts.
Change-Id: I4f2a491d830550576b285a2a7df6c89cc93c2312
Reviewed-on: https://go-review.googlesource.com/38384
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This renames some types to be exported (automated using gorename), and
then adds a package "godata" to get the Go project's current
maintner.Corpus (currently a stub implementation). It then adds the
start of a "gostats" command using "godata", moving some temporary
stuff from maintnerd in the process.
Change-Id: I31f7d7af6f60892317a9f55ae44106543bfb599d
Reviewed-on: https://go-review.googlesource.com/38377
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
More work on syncing issue events.
This adds a new GithubIssueEvent proto type, adds ways to go both from
and to the in-memory githubEvent type, and cleans up the event tests
added in 8927fdda9.
The next commit will wire it up to the syncing.
Change-Id: Ibdd9759dab432c6eaa4c37becfbb31f1ebf8f5b9
Reviewed-on: https://go-review.googlesource.com/38370
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
More remains, but this is as good of a checkpoint as any.
Change-Id: I63f6b13508562143159e92d78fc4ef60070342ae
Reviewed-on: https://go-review.googlesource.com/38163
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Currently we set GOROOT in both AddGoGit and in PollGitCommits, which
seems like a mistake - the latter should use whichever directory has
been specified in the configuration.
Fix an error in a function title.
Change-Id: I7447d06a6914b68802ebfc5a19230a39b9b9a776
Reviewed-on: https://go-review.googlesource.com/38233
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
One of the recent commits changed the nil value for Assignees when
a new githubIssue gets created.
I used github.com/kr/pretty to determine the diff after the
reflect.DeepEqual failed; it's pretty useful and would be handy to
have every time, though it would be another dependency to add.
Change-Id: I5d71296a6caf1776fd95c9991d798dc848152aaa
Reviewed-on: https://go-review.googlesource.com/38162
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
And add a dummy query for fun.
Change-Id: Icc3f846691b83433f4b47d68b0a6c102be275ae9
Reviewed-on: https://go-review.googlesource.com/37994
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
No functional changes, just moving Github related code to its own
file.
Change-Id: I9600518142202c5519984564743a1996615d693a
Reviewed-on: https://go-review.googlesource.com/37992
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This is just the start of git commit metadata importing, but without
any of the git-specific bits. This is just the proto changes and the
changes to wire up the guts with maintner. The guts will come in a
following CL.
Change-Id: I321590dc30ba5c62346911526fed647fc0d0b3ad
Reviewed-on: https://go-review.googlesource.com/37938
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Due to the design of the github paging API it's possible to process
messages twice in the same paging run, which can lead to early
termination since the second run of the message will have the same
"UpdatedAt" as the first time we saw it. Keep track of messages we've
seen before in this paging run and ignore them the second/third times
we see them.
Also skip subdirectories of the `--data-dir` argument; we don't want
to search any other directories for data files than the one provided.
Change-Id: Ib4b867cc50568f7bfc5a3d3a0158e284d2038808
Reviewed-on: https://go-review.googlesource.com/37794
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This should comprise most of the fields that are needed by the
existing godash/devapp app. Will wire up the Milestone code in
another commit.
Change-Id: I83c399afebce5865dd03c00e295be5a19e1e36b2
Reviewed-on: https://go-review.googlesource.com/37753
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Eventually we will add other mutation sources but for the moment
let's catch up from the same proto directory we are writing changes
to.
Change-Id: I6efa3c0fb6f49cf4240d8a365f558bbb1c52602e
Reviewed-on: https://go-review.googlesource.com/37733
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
If the context returns, immediately return from the function instead of
continuing to sleep. Also, remove deprecated oauth2.NoContext.
Change-Id: If92c18d9d4dc1a801be8df5ad2e78ae809b9f274
Reviewed-on: https://go-review.googlesource.com/37650
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
For now, open and close the file each time we write a protobuf. We can improve
the performance in the future as necessary.
Change-Id: Idcc75df5aa59a604c0c0aeeff6ea0e0ff0f2db8f
Reviewed-on: https://go-review.googlesource.com/37626
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
When we get an issue from Github, add the issue and associated
details to the Corpus. Check whether the `Updated` timestamp on an
issue is newer than the one we have in the Corpus; if the timestamp
is the same, we can stop processing since we are sorting by the
updated date and no older issues will have new data.
If the mutation changes the Corpus, add it to an array of Mutations
that we will eventually flush to disk.
Adds a helper function to convert a github.Issue into a
maintpb.Mutation - this way we can reuse the logic for adding data to
the Corpus. Adds some more tests around data processing.
Change-Id: I9475e293941784179420ca7b638917943be1f29e
Reviewed-on: https://go-review.googlesource.com/37490
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Now you can actually pass repos to maintnerd and tell a Corpus to
track/store them. Adds a hook to "catch up" a given repo, though it's
unimplemented at the moment, so all requests hit the API. Also only
Github is implemented.
Change-Id: Ib074c02437d6530fc8d57dd433486c05cd75ee64
Reviewed-on: https://go-review.googlesource.com/37299
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The latest master for go-github (google/go-github@23d6cb9c) adds
Context parameters as the first argument to every function call.
Update code that calls go-github to have contexts, where appropriate,
and adds timeouts to some long-running commands that didn't
previously have them.
Fixesgolang/go#19208.
Change-Id: I25203de5d10ada1dcd3a97eb5434a85bb328ce7e
Reviewed-on: https://go-review.googlesource.com/37293
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
WIP; not functional yet, but submitting so
Kevin can continue if he wants.
Change-Id: I1dab38f9845ebcbd69ec2a6f07e98793d3ed7cfb
Reviewed-on: https://go-review.googlesource.com/37053
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Add $GOPATH/src to the --proto_path so we can import the google
Timestamp definition instead of redefining a copy in our tree.
Change-Id: I09c3554fb025a32c9db4892b702ee2bedb3c0e53
Reviewed-on: https://go-review.googlesource.com/36899
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This will be a package and server for working with Git, Github, and
Gerrit data.
Change-Id: I12c04d31c792d6cf752ce63b05a0b155d1f53f26
Reviewed-on: https://go-review.googlesource.com/36554
Reviewed-by: Kevin Burke <kev@inburke.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Kevin Burke
Brad Fitzpatrick