This is a refactor to bring the redirect check into a small dedicated
function within gosrc. It can be tested more directly/easily.
Change-Id: Ib9c907bae00384c1f7fef74146b19b715976c546
Reviewed-on: https://go-review.googlesource.com/134475
Reviewed-by: Tuo Shan <shantuo@google.com>
This change fixes a regression from CL 120059. We started to redirect
to GitHub's reported canonical case when the request path didn't
match too aggressively. It should only be done if the request path is
already on GitHub; otherwise we're redirecting a vanity import path to
its GitHub hosting location whenever it doesn't have an import path
comment.
Add tests for this newPackage redirection behavior to increase
confidence in the fix, and help avoid future regressions.
Fix typos in gosrc package comments.
Fixesgolang/gddo#579.
Change-Id: I7ed0787a125d2f39ca979d01983e971729e181fd
Reviewed-on: https://go-review.googlesource.com/134355
Reviewed-by: Tuo Shan <shantuo@google.com>
Support for the context package was added in baa18d8, but the
gosrc/print.go debug utility was never updated, and since it has the
ignore build tag go test etc. didn't catch this.
This will not affect godoc.org operation; it's just a debug utility for
developers; it was helpful for me in examining #576.
Change-Id: Ie48ede8b1a54289d5dd848c4ca7867fd9ca38a02
GitHub-Last-Rev: ad319e617b
GitHub-Pull-Request: golang/gddo#577
Reviewed-on: https://go-review.googlesource.com/132655
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Go 1.11 adds an experimental WebAssembly port. The port uses
new GOOS=js GOARCH=wasm values. Some packages, such as syscall/js,
are implemented only for js/wasm environment, so this needs to be
one of the environments to try in order for its documentation to
be available.
Updates golang/gddo#572.
Change-Id: I35b6eae735602e7387beacff07f2b7d6fe323314
Reviewed-on: https://go-review.googlesource.com/131343
Reviewed-by: Tuo Shan <shantuo@google.com>
This change regenerates the list of standard packages to contain
packages that are new to Go 1.11, which has been recently released.
It was done by running:
go generate ./gosrc
Updates golang/gddo#572.
Change-Id: Iaefbd8e7119b9bfb46944e137da652c6209e7231
Reviewed-on: https://go-review.googlesource.com/131342
Reviewed-by: Tuo Shan <shantuo@google.com>
Problem:
Our GitHub organisation is named Teamwork, but for various hard to
change reasons we import packages as teamwork (lower case t). To enforce
this we add an import enforcement for our packages:
package duck // import "github.com/teamwork/duck"
This works well, but godoc.org enforces users to use the canonical
GitHub path (Teamwork), but with the import statement the Go compiler
enforces a different path, leading to an error.
Fix:
Instead of immediately redirecting in the gosrc package, it will now
only store the canonical import path there. This is before the Go source
code is actually scanned, so we don't *actually* know what the real
canonical path should be.
In the doc.newPackage() function it will check for the canonical import
path, taking both the Go import path as well as the reported path from
the gosrc package in to account, and redirect as needed.
This seems to work for all the cases I can think of:
github.com/Teamwork/validate -> github.com/teamwork/validate
github.com/pkg/ERRORS -> github.com/pkg/errors
github.com/Carpetsmoker/sconfig -> arp242.net/sconfig
arp242.net/SCONFIG -> not found (expected)
bitbucket.org/pkg/inflect -> works
bitbucket.org/pkg/INFLECT -> works (should probably redirect too)
github.com/docker/docker/client -> works (#534)
github.com/moby/moby/client -> redirects (on master this actually seems to error out?)
It should also be easy to add a similar check for for some other repo
providers, such as BitBucket, GitLab, etc.
Fixes#507
Change-Id: I49c5ccb328694f95dd7a0e5d577297d56d88893f
GitHub-Last-Rev: 28efc37564
GitHub-Pull-Request: golang/gddo#560
Reviewed-on: https://go-review.googlesource.com/120059
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This reverts commit 9b12a26f3f (CL 90855).
CL 90855 applied a change to fix issue golang/gddo#402, but caused a
regression golang/gddo#533. The regression led to an earlier version
of gddo to be redeployed.
There was an attempt to resolve issue golang/gddo#533 in CL 93196,
which was later picked up and resent as CL 130496.
Tuo and I reviewed CL 130496 and its behavior, and found that
it was not a satisfactory resolution. It introduced additional
complexity and was better in some contexts and situations, but
worse in others. The main downside is that it made it less clear
why some internal packages were being hidden in some places but
not other places.
To fully resolve the original golang/gddo#402, we will need to
come up with and evaluate a more complete plan of action. Until
then, we want to restore the source code to a state which can be
deployed, so take the short term action of reverting the original
CL and go back to a simple and functional behavior: always display
all packages.
Updates golang/gddo#402.
Fixesgolang/gddo#533.
Change-Id: I8fb6ac05aa6f118e740e27eaec1bcf2fc9d32766
Reviewed-on: https://go-review.googlesource.com/130937
Reviewed-by: Tuo Shan <shantuo@google.com>
Currently, gddo assumes that there will be just one go-import HTML meta
tag when resolving a custom import's VCS repository. vgo introduces a
special new go-import meta tag to identify the location of published Go
modules; it has the VCS identifier 'mod'.
https://golang.org/issue/25140 and https://golang.org/issue/25139 added
minimal awareness to the Go 1.9 and 1.10 branches for the vgo transition
that will start in Go 1.11. Part of this backported support is to ignore
the module special go-import meta tag which has a 'mod' vcvs type:
https://go-review.googlesource.com/c/go/+/115298/4/src/cmd/go/internal/get/discovery.go
Per https://golang.org/issue/25069, the 'mod' vcs type is used by 'new'
Go code to identify where published modules can be fetched.
As a first minimal step we can make gddo "aware" in the same way that Go
1.9 and 1.10 are "aware" of these published modules by simply ignoring
that go-import meta tag.
Later gddo CLs will likely need to enhance support for vgo, but this is
a sufficient first step.
Fixes#558
Change-Id: Ibddfcc8e0a663792da206a244e5cffb8c68fe894
Reviewed-on: https://go-review.googlesource.com/120817
Reviewed-by: Tuo Shan <shantuo@google.com>
This was accidentally left behind in abcff89 during review.
Change-Id: Ia259269a56e37754f4faf793b13deb9a918359a1
Reviewed-on: https://go-review.googlesource.com/114975
Reviewed-by: Tuo Shan <shantuo@google.com>
Currently, the server sends a 404 when the URL contains query parameters
unknown to the application. Instead, ignore the parameters. Fixes#500
Change-Id: I4478118311ee57c2f4adcb28fcdfff98357d0c39
GitHub-Last-Rev: f2754463e0
GitHub-Pull-Request: golang/gddo#548
Reviewed-on: https://go-review.googlesource.com/103400
Reviewed-by: Dmitri Shuralyov <dmitri@shuralyov.com>
Reviewed-by: Tuo Shan <shantuo@google.com>
To prevent leaking sensitive data in the normal case, use the Debug
logging level instead of Info when logging credentials.
There are very sensitive pieces of data in the settings, namely GitHub
secrets, Redis database passwords, etc. In the normal case, these are
logged. To prevent accidental leaking of credentials, only print
settings map when logging in Debug mode.
Change-Id: I4144074bf2b26a940fd9bfb4ed695b9112e67521
GitHub-Last-Rev: 70b36f7481
GitHub-Pull-Request: golang/gddo#545
Reviewed-on: https://go-review.googlesource.com/102278
Reviewed-by: Parker Moore <parkrmoore@gmail.com>
Reviewed-by: Tuo Shan <shantuo@google.com>
Update references of go-talks.appspot.com to talks.godoc.org, because
the talksapp is hosted on the talks.godoc.org domain now and the old
appspot domain is irrelevant.
Also, some of the references are incorrectly go-talks.appspot.org which
must be a typo, because that domain displays a for sale sign and does
not appear to be owned by Google.
Fixesgolang/gddo#532
Change-Id: I8b8ec58e2f2de0e1f2f8d66a6adedc2cb50da754
Reviewed-on: https://go-review.googlesource.com/95256
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Only redirect to canonical github project when the requested is a case
insensitive match, but not an exact match to the requested project.
Otherwise projects which have been moved to a new name, but use a
canonical import to the old name will result in a redirect loop.
Change-Id: I350359d802c3c42e47f2247ffee2469d6449eabc
Reviewed-on: https://go-review.googlesource.com/93176
Reviewed-by: Tuo Shan <shantuo@google.com>
Add redirects from HTTP to HTTPS. Add HSTS header. Update example URLs
to use HTTPS.
Fixesgolang/gddo#536
Change-Id: Iffbf1cff8ed0e87a8cae90496907577f410e4444
Reviewed-on: https://go-review.googlesource.com/93276
Reviewed-by: Tuo Shan <shantuo@google.com>
List internal packages only underneath their direct parent
or whether is browsing the internal package themselves.
Change-Id: I856178b427de2486c17b8f162e64be62bded9272
Reviewed-on: https://go-review.googlesource.com/90855
Reviewed-by: Tuo Shan <shantuo@google.com>
0 to automatically pick a port.
Change-Id: I6619af41ed18520f1c39256407fa774a660fd513
Reviewed-on: https://go-review.googlesource.com/86076
Reviewed-by: Ross Light <light@google.com>
Allow user to set any minimum value for wait time > 0.
Change-Id: I61592707f840029c6f401e383d66fd9608b993f5
Reviewed-on: https://go-review.googlesource.com/85655
Reviewed-by: Ross Light <light@google.com>
Newer Redis version requires arguments types to be string or integer.
Change-Id: I5f3fba9affda3b0747372da824ca0662cc95a495
Reviewed-on: https://go-review.googlesource.com/78535
Reviewed-by: Ross Light <light@google.com>
httptest.Server.Client was added in Go 1.9, and we want to support Go
1.8+ for App Engine.
Change-Id: I1eb539a69c2dadc85505b181eeeb6e20dd333df7
Reviewed-on: https://go-review.googlesource.com/78539
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Adding initial prototype of runtimeconfig package which provides ability
to be notified of runtime config changes from Cloud Runtime Configurator
service.
Change-Id: Id5765e15bc960fd001286ee14678c5ff8e2ec168
Reviewed-on: https://go-review.googlesource.com/69297
Reviewed-by: Ross Light <light@google.com>
1. Use cmp package to replace reflect.DeepEqual
2. db.PutIndex and db.DeleteIndex won't return error when remote_api is
not setup, since they are only called inside db.Put and db.Delete.
db.Search remains unchanged since it is called externally.
Change-Id: Ife84f5a98abe721fbb127e497fdae34125d547a1
Reviewed-on: https://go-review.googlesource.com/78435
Reviewed-by: Ross Light <light@google.com>
Currently only serving Redis, but could be used for other GCP gRPC
connections.
Mildly surprising behavior: taking down Redis does not make Pool.Get
return an error: it isn't until sending another request on a connection
before the pool starts failing. This only matters when testing the
health checks with no load.
Change-Id: Ie720b80a398fd9f7d4aa6ab0c6a88adaa85ccdc9
Reviewed-on: https://go-review.googlesource.com/76750
Reviewed-by: Tuo Shan <shantuo@google.com>
Results from my workstation:
BenchmarkE2E/Baseline-12 20000 85744 ns/op 5171 B/op 68 allocs/op
BenchmarkE2E/WithLog-12 20000 98693 ns/op 6321 B/op 102 allocs/op
We can expect ~10us CPU and ~1KB RAM of overhead per each request.
Likely fluentd will be the bottleneck at that point.
I dug in with the profiler and the overhead is almost entirely coming
from JSON encoding. If performances becomes an issue, use of the
encoding/json package could be eschewed for a custom marshaler.
Change-Id: I757740b31d168ac339e4775bf24212ba12329ae9
Reviewed-on: https://go-review.googlesource.com/73770
Reviewed-by: Sarah Adams <shadams@google.com>
- Request/response body sizes are collected on a streaming basis
(negligible overhead).
- Header sizes are approximate and only based on what the underlying
handler sends or receives. net/http may add extra headers, so headers
will likely be undercounted. I don't see a reliable way to address
this.
- Includes a fluentd logger to send the forward input plugin that is
open by default in google-fluentd. I've integration tested this
middleware locally. Unit tests verify that the format is correct.
- Hijack/Websocket support is missing. This is fine for gddo.
Change-Id: I53c006edc1c3004e08c1b7e4b68fa94240f38db6
Reviewed-on: https://go-review.googlesource.com/70790
Reviewed-by: Herbie Ong <herbie@google.com>
Each of the servers has slightly different needs for configuring their
outbound HTTP transport. A little copying is better than a little
dependency.
Notably, the configuration logic in gddo-server is now consistent with
the rest of the configuration-gathering logic.
Change-Id: Ibe7d287a102e20e6e13d97e8a6483d9febab6442
Reviewed-on: https://go-review.googlesource.com/69291
Reviewed-by: Tuo Shan <shantuo@google.com>
This uses Stackdriver trace API to trace search, package requests
as well as background tasks. It also traces outbound requests to
VCS's and remote_api calls.
Change-Id: Ice82c1fbd267eefa4bad9d6674dc39ef10a1c874
Reviewed-on: https://go-review.googlesource.com/68472
Reviewed-by: Ross Light <light@google.com>
Might just be a hypothetical concern, but better for this to fail closed
(not send credentials when needed) than to fail open (send credentials
insecurely).
Change-Id: I8542e8c0b533a3c5f53a5f03f4e3b639b6b55f71
Reviewed-on: https://go-review.googlesource.com/69290
Reviewed-by: Tuo Shan <shantuo@google.com>
This is intended to simplify the number of stages of initialization in
gddo-server. Before this CL, globals and the config may or may not be
correctly initialized at various points of executing methods during
main(). While this largely does not come up because the handlers are
not used until appengine.Main, it can be problematic for background
tasks. After this CL, handlers (being methods on the server struct) can
depend on server being completely initialized, and the server can depend
on the config being completely initialized.
Change-Id: I1a83a2337a2cdb4d98ec00b26d5e0498ed7e4edd
Reviewed-on: https://go-review.googlesource.com/67290
Reviewed-by: Tuo Shan <shantuo@google.com>
This is in preparation for passing around the template map explicitly
instead of using it as a global. It also moves the manifest of template
sets into templates.go and out of main.go.
Change-Id: Ic7db8d5ab0186c1f4eb2446871e2c18ae1554d82
Reviewed-on: https://go-review.googlesource.com/67051
Reviewed-by: Tuo Shan <shantuo@google.com>
Remove the gaAccount global variable in favor of reading from
configuration.
Minor behavior changes:
- The GA_ACCOUNT environment variable will override the GCE metadata
ga-account, where before it was not being read on GCE.
- The project ID across the program defaults to the one from GCE
metadata, overridden by the GCLOUD_PROJECT environment variable or the
command-line flag. The GCE logger now uses this project ID instead of
just considering the GCE metadata.
- The GCE log name can now be set using the gddo_gce_log_name
environment variable.
Change-Id: I0ed2ae7a5130d63146e9bf2746f22980c8a782b9
Reviewed-on: https://go-review.googlesource.com/67050
Reviewed-by: Tuo Shan <shantuo@google.com>
There is much more complicated behavior in isRobot based on usage by IP
address. This isn't being tested, but requires access to the database.
This is in need of a fuller test (as is much of the gddo-server
directory).
Change-Id: I4f3467b30e3ecc83bf72712f01a183bd211adc16
Reviewed-on: https://go-review.googlesource.com/66931
Reviewed-by: Tuo Shan <shantuo@google.com>
These tasks operate concurrently with the request flow anyway, so each
task should be on its own tick loop.
Change-Id: I129212b297db15395f7852e5b73e66081c63fbd0
Reviewed-on: https://go-review.googlesource.com/66652
Reviewed-by: Tuo Shan <shantuo@google.com>
Dmitri Shuralyov
Martin Tournoij
Dmitriy
Tuo Shan
Paul Jolly
Mike Kabischev
Suriyaa Sundararuban
David Symonds
Frederik Ring
Parker Moore
Andrew Bonventre
Leigh McCulloch
Daniel Nephin
JBD
Herbie Ong
Ross Light