When a URL path for a package page fails the check for
module.CheckImportPath, redirect to the homepage. Otherwise, the page
will 400 on pkg.go.dev.
Change-Id: Ic337b968a3919b2e4fcd0ab4dd6b76c631f5edf9
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/279035
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Requests to github.com/golang/go and github.com/golang/go/src are now
redirected to pkg.go.dev/std.
Requests to github.com/golang/go/src/<path> are redirected to
pkg.go.dev/<path>.
Change-Id: Iaf9f52d17fc19c3e6c325c260de3f070da32a21b
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/279036
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When a godoc.org requests results in a 200, but a 404 on pkg.go.dev,
make a fetch request to pkg.go.dev to fetch that path and log the
output.
Change-Id: I1ca12732317d7e90b42eff022357067d21f84f9b
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/279038
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Code used to tee traffic from godoc.org to the pkgsite teeproxy is
deleted, since requests are now teed directly to pkg.go.dev.
Change-Id: Icf89a310afec7cb225e5e88be6b1265fc4bee4f5
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/279037
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gddoEvent was incorrectly renamed to requestEvent in CL 274860, which is
now fixed.
Change-Id: I16b2cb483994fbe3a2adddf800fcf712b31c54b4
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/279034
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Text on the right side of the banner is bolded to emphasize
"Always use pkg.go.dev".
Screenshot:
https://photos.app.goo.gl/nNeRP8zpwi4jGuGk6
Change-Id: I97a7a6b10a5710810e9a9c652d8012492f3e4d95
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/275872
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Trust: Julie Qiu <julie@golang.org>
The logic for PkgGoDevURL is updated:
- The ?tab=doc query param is removed
- The ?tab=packages query param is removed
- Requests for the godoc.org badge SVG are teed to the pkg.go.dev badge SVG
- Requests to godoc.org/C are explicitly teed to pkg.go.dev/C
Change-Id: If5bd6259a88e921b8175e993ea574ea47947e1ca
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/274859
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Trust: Julie Qiu <julie@golang.org>
When a request is expected to 404 even after we redirect godoc.org to
pkg.go.dev, don't tee that request.
Change-Id: I1640bb2886ccdd55b2790f3b8d58b2d32eca8418
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/274699
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Request to /-/subrepo are not redirected to
pkg.go.dev/search?q=golang.org/x instead of pkg.go.dev.
Change-Id: I582271b112a0ce482bedfd822523cf7e4dbc9059
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/274697
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Previously, the godoc.org request status was not being sent to the
teeproxy. This is now fixed.
Change-Id: I90c33ee3fe0f75cbbff42b5d6b16b29bd3a1dd2d
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/249961
Reviewed-by: Jonathan Amsterdam <jba@google.com>
With golang.org/cl/242097, the teeproxy no longer needs
the RedirectHost field from gddoEvent.
Change-Id: Iaf1647d7939084b2bf2d41816d259c7c274283dc
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/242098
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Pkg.go.dev does not support documentation for vendor directories.
When showing the corresponding pkg.go.dev link in the header or
when automatically redirecting, change the destination to be the
pkg.go.dev homepage instead of a vendor link that will 404.
Fixesgolang/go#38671
Change-Id: Ic7d221d3cc7fef352f5c7097a667fb8dd2ad0340
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/233839
Run-TryBot: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
Most browsers send requests to http://godoc.org/net/http as
GET /net/http HTTP/1.1
Host: godoc.org
instead of as expected in current tests
GET https://godoc.org/net/http HTTP/1.1
Host: godoc.org
This causes http.Request.URL to have empty strings for the "Scheme" and
"Host" fields (see https://golang.org/src/net/http/request.go#L120 and
RFC 7230 Section 5.3). As a result, server logs show most gddoEvent
structs as having an empty "Host" field and a relative path for the URL.
This CL enables the parsing of these fields from the other fields of the
http.Request struct.
Change-Id: Iab923a0d9a70f32f36b6e4a15019afc648699796
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/236143
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This boolean field will reflect whether the godoc.org request was
redirected to pkg.go.dev.
Change-Id: I521c98e7342f5880021bd215640adb570935cb06
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/235957
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Currently, requests to App Engine admin paths at godoc.org/_ah/* are not
redirected to pkg.go.dev.
This change stops making a POST request to the teeproxy when a user
visits any of the admin paths.
Change-Id: I77f39b8289eab1086bc987adb218ee55ed48c919
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/235519
Reviewed-by: Julie Qiu <julie@golang.org>
This will indicate whether the request to pkg.go.dev was made by a
robot.
Change-Id: I64931b9ff8a4da575d0614771a0575f2bd0746a0
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/235448
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
There should be an upper bound for the number of "dot" processes
that can be started concurrently in order to serve incoming HTTP
requests for the ?import-graph pages, to help avoid resource
exhaustion and server overload.
This change implements a limit of 10 such processes at once.
Change-Id: I65d01996461ae212bb84cfd8edfbe6fcec43c329
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/234637
Reviewed-by: Robert Findley <rfindley@google.com>
This will reliably indicate to pkg.go.dev that a request is due
to the automatic redirect logic.
Additional logic is added for more accurate mapping of
godoc.org -> pkg.go.dev URLs.
Updates golang/go#37099
Change-Id: Iba380a3b5951d50576c17711990d0ba87857b926
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/224659
Reviewed-by: Julie Qiu <julie@golang.org>
This change adds UI for the option to set a cookie and have
candidate godoc.org links redirect to their equivalent pkg.go.dev
counterparts. If a user returns to godoc.org with the utm_source
form value equal to 'backtogodoc', it will display an option to
turn off the automatic redirect.
The intention is to have a link on pkg.go.dev that allows the
user to return to godoc.org and also turn off the automatic redirect
without having to know the special URL param values.
Updates golang/go#37099
Change-Id: I27bd03cbc484a1e504795ff669224c2a2a7d72b6
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/222315
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
At the end of a request to *.godoc.org, an additional request is made
to the teeproxy for pkg.go.dev containing data about the request.
These requests are used to collect data about godoc.org, and ensure that
all incoming traffic to godoc.org can be handled appropriately by
pkg.go.dev.
Change-Id: Ib0b92b81c2a361f5b6a1813ddc62d090edbe4ba1
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/219017
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
+ The go111 runtime requires google.golang.org/appengine to be
updated to v1.6.5 in order for legacy APIs to continue to work
+ Small changes need to be made to the main package so that there
is a proper main function
Steps run to update vendored dependencies:
go get -d google.golang.org/appengine@v1.6.5
go mod tidy
go mod vendor
Change-Id: Ibfec53f00e8ba4874c36478607f68b0e652e09f6
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/216298
Reviewed-by: Bryan C. Mills <bcmills@google.com>
In order to ensure that full-text-search keeps running,
the google.golang.org/appengine package needs updating.
The first step of this is to convert to using modules
before updating the dependency.
Commands run:
go mod init github.com/golang/gddo
go mod tidy
go mod vendor
rm -rf Godeps
Change-Id: I6b5d1465226217bb1c0f134c90cd94aa2e7046d1
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/215678
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
In order to better understand what endpoints are being used
on godoc.org, an entry is now logged for the start and end of each
request. The latency of each request is also logged.
Change-Id: I8198b1206d04564e86b782e3e94964e829d5cc9b
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/209223
Reviewed-by: Tuo Shan <shantuo@google.com>
The golang.org website has changed its UI design, and no longer
includes "Build version <version>" text at the bottom of each page.
The build version is still specified on each page in a short script.
Use that instead.
Updates golang/gddo#644
Change-Id: Iedac4fa1f44510af6462f9bb3431c5c23eaaed16
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/193258
Reviewed-by: Tuo Shan <shantuo@google.com>
By now, there are some large packages whose documentation exceeds
the current size limit of 800 KB. Increase the limit by 1.5x to
accomodate such packages.
Fixesgolang/gddo#635
Change-Id: Ie846366d2aea6a58f3cca3080c99fc0124995e5d
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/190398
Reviewed-by: Tuo Shan <shantuo@google.com>
We were correctly handling the error where the file did not exist by
mapping it to NotFoundError, which in turn was served as a normal 404
page to users. We were not handling the error where the file did exist
but was not a directory, which led to user-facing 500 errors.
This change fixes that by adding a check that the opened file is a
directory before attempting to read its contents.
Also, the file that was opened via os.Open was never closed. Fix that.
Fixesgolang/gddo#640
Change-Id: I2b0cee6431a81294757b6fd24191fdd27f41ef63
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/190397
Reviewed-by: Tuo Shan <shantuo@google.com>
Bitbucket Cloud REST API version 1 is deprecated effective
30 June 2018. All 1.0 APIs will be removed from the REST API permanently
on 29 April 2019.
Since these are APIs used in gosrc/bitbucket.go, this CL aims to migrate
to the version 2 APIs. Most things translated well: all references to
/1.0/ were replaced with /2.0/ and json.Marshal structs were updated.
Note, many v2 APIs replace maps with arrays.
Pagination is present in many v2 APIs requiring iteration of some APIs:
refs, watchers, and src. pagelen has been set to max, 100, to minimize
number of API calls, but this could lead to context timeouts.
Timestamp formats have moved to time.RFC3339, thus all bespoke layout
vars have been removed.
/2.0/repositories/{username}/{repo_slug}/refs can directly fetch both
branches and tags, thus that logic has been collapsed.
/2.0/repositories/{username}/{repo_slug}.parent takes the place of both
/1.0/repositories/{username}/{repo_slug}.is_fork and
/1.0/repositories/{username}/{repo_slug}.fork_of. It is left as an
interface{} since the contents are not used.
gosrc.bitbucketRefs.Values[].Target.Hash, previously
gosrc.bitbucketNode.Node, stores the full SHA1 hash, not the first 12
chars. It does not appear to break anything to use the full hash, so
that is done.
/2.0/repositories/{username}/{repo_slug}/src interleaves files and
folders, so new processing logic was required.
/1.0/repositories/{username}/{repo_slug}/raw has been deprecated in
favour of directly using /2.0/repositories/{username}/{repo_slug}/src.
Followers, now called watchers, and are not accessable via the root
/2.0/repositories/{username}/{repo_slug} API. They have their own
paginated api, /2.0/repositories/{username}/{repo_slug}/watchers, for
fetching detailed user information. Enumerating watchers will require at
least one API call, and testing indicates that this API does not agree
with the value of /1.0/repositories/{username}/{repo_slug}.followers_count
or the Web UI, which agree with each other. While this feels like a bug
on Bitbucket's end, it was decided that gosrc.Directory.Stars should be
left default. The godoc for gosrc.Directory has been updated to reflect
this change.
Fixesgolang/gddo#599
Change-Id: I8c471efb32a49c9aae179f7174e018357f93b9df
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/170279
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
The db.DeleteIndex method checks that db.RemoteClient is not nil
before dereferencing it. The direct call to deleteIndex did not,
and therefore panicked when db.RemoteClient was nil.
This makes Database.Block method consistent with Database.Put and
Database.Delete, which also use the db.DeleteIndex method.
With this change, go test ./database passes.
Change-Id: I90514bc1e855e125a9db57a93d2d7a88bce67e2e
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/172961
Reviewed-by: Tuo Shan <shantuo@google.com>
The method was used in the gddo-admin block command. It adds the URL
to the blocked list and removes everything under that URL from the
database. To make it useful, this change should also remove anything
under the URL from the newCrawl queue and the App Engine search index.
Change-Id: I02020801556418616f6628dca9afa39db09ad88f
Reviewed-on: https://go-review.googlesource.com/c/gddo/+/130756
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change regenerates the list of standard packages to contain
packages that are in Go 1.12, which has been recently released.
It was done by running:
go generate ./gosrc
Updates golang/gddo#603
Change-Id: I641eeb60c5a47feaa93dc1620ee9ad96ed6878ca
Reviewed-on: https://go-review.googlesource.com/c/164301
Reviewed-by: Tuo Shan <shantuo@google.com>
Go 1.8 isn't supported anymore by goapp test (which is run by Travis CI).
Also add deploy command to README.
Change-Id: I90ee9a699fd95c74ce51446dbc35533738c05eea
GitHub-Last-Rev: af88a3b3eb
GitHub-Pull-Request: golang/gddo#590
Reviewed-on: https://go-review.googlesource.com/c/150059
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Francesc is no longer actively maintaining this project.
Also improve the GitHub issue tracker link to prepopulate
the "talksapp: " issue title prefix.
Change-Id: Id3b337039734e59c792310796b3facbd979d7909
Reviewed-on: https://go-review.googlesource.com/c/140677
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Tuo Shan <shantuo@google.com>
Reviewed-by: Francesc Campoy Flores <campoy@golang.org>
The current implementation does not resolve background images, therefore
they do not display on talksapp. This change handles .background in the
same way as .image and .iframe.
Fixesgolang/gddo#541
Change-Id: I309688fb95bc407fb59e8eb4c5e5ea2f3c54b184
Reviewed-on: https://go-review.googlesource.com/c/133235
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Julie Qiu
shaquilleq
Andrew Bonventre
shaquilleq
Dmitri Shuralyov
Gopher
Colin Arnott
Tuo Shan
Harald Nordgren
Michael McLoughlin
Stuart Jansen