Change-Id: Ibc1eb122a634fed43f6d7928545ccf3a0232515c
Reviewed-on: https://go-review.googlesource.com/25461
Reviewed-by: Dmitri Shuralyov <shurcool@gmail.com>
Reviewed-by: Dave Day <djd@golang.org>
The logging client does not rely on the legacy cloud context behaviour
(it uses the project ID explicitly passed into the NewClient call), and
the cloud.NewContext function is due to be deleted.
Change-Id: I5a58dc0b8ab25d942a13b0386bccc7f3b2726b33
Reviewed-on: https://go-review.googlesource.com/25418
Reviewed-by: Andrew Gerrand <adg@golang.org>
This CL blocks the access to import-graph pages for robotRequest, which
causes the recent outage since graphs are very expensive for CPU and
disk read I/O. This is a hot-fix that has already been deployed. We'll
work on something permanent later when we come up with some better plan
for robotRequest related issues.
Change-Id: I7e7fbc0538f3d8d26c1ca35cef51e828a79dc1c9
Reviewed-on: https://go-review.googlesource.com/25416
Reviewed-by: Andrew Gerrand <adg@golang.org>
Use app engine's main function to start the server by default. GAE
search is also enabled by default. API search is also using the new
search algorithm.
Developers will need to install Google Cloud SDK to use dev_appserver
in order to run the search api locally. The development setup will be
updated as soon as this CL is submitted.
Change-Id: Idae7a88949b2757977d0ab33628564451a1dddd3
Reviewed-on: https://go-review.googlesource.com/25360
Reviewed-by: Andrew Gerrand <adg@golang.org>
Reviewed-by: Alan Donovan <adonovan@google.com>
gddo should let users visit any go-gettable package. This change will
remove those noise packages from the search index but not the db.
This fixes#420
Change-Id: Ifb1d731655ca5e6d31bfc0bd4c7657ec165b9902
Reviewed-on: https://go-review.googlesource.com/24793
Reviewed-by: Andrew Gerrand <adg@golang.org>
Reviewed-by: Alan Donovan <adonovan@google.com>
We are about to reindex the production search index. Update the version
number to force the crawler to crawl all packages without checking the
existing Etags.
Change-Id: I12d865188e23933344becb4449b37385a3e59a1a
Reviewed-on: https://go-review.googlesource.com/24701
Reviewed-by: Alan Donovan <adonovan@google.com>
A package will be removed if either it has no commits in two years and
no imports from other packages, or it is created for a quick bug fix,
which means it has one or two commits within a week of creation time and
no other activity since then.
This CL checks such packages from GitHub and BitBucket utilizing their
API to gather commits information.
This CL also removes the checks for references for GitHub repo, instead
it check the most recent commit from default branch.
This fixes#405
Change-Id: I14b0f0133f31851511aaa63eee8acbfba63e13d2
Reviewed-on: https://go-review.googlesource.com/24513
Reviewed-by: Alan Donovan <adonovan@google.com>
database.PackageDocument is now merged into database.Package.
database.Package now implements search.FieldLoadSaver, which can
customize the Rank method to our default sorting algorithm. This allows
faster search queries by avoiding sorting. This also eliminates the
confusion of having two similar package structs in database.
Two admin handlers are used to reindex and purge index. These handlers
are restricted to be accessible to admin only in the yaml file.
Change-Id: I63929a50b08b9817e7d62020f9755fba9d4ec8f0
Reviewed-on: https://go-review.googlesource.com/24450
Reviewed-by: Alan Donovan <adonovan@google.com>
Reviewed-by: Andrew Gerrand <adg@golang.org>
The data we have now are still Go 1.6.1, which causes gddo not
recogonizing some of the std packages.
Change-Id: Ic06e11ac0b04aa2a60cd9376b70a637099bce9cf
Reviewed-on: https://go-review.googlesource.com/24432
Reviewed-by: Alan Donovan <adonovan@google.com>
The memcache underneath is using the gomemcache package, which can be
used both as a standalone app or in App Engine. The cache will reduce a
significant amount of HTTP requests we make when updating packages. Most
of the requests are to GitHub. So this change will save a lot against
GitHub rate limits.
Change-Id: I46c7c4bd658b7b2ab18ca67e8c8d25549181dfb4
Reviewed-on: https://go-review.googlesource.com/24326
Reviewed-by: Alan Donovan <adonovan@google.com>
Reviewed-by: Andrew Gerrand <adg@golang.org>
Prevent 'govet' from complaining about copying
a lock value ("httputil.CacheBusters contains sync.Mutex").
Change-Id: If2d235f148e767ad0a9e6110ae1f38e69a32b1a9
Reviewed-on: https://go-review.googlesource.com/24260
Reviewed-by: Andrew Gerrand <adg@golang.org>
Store and show if a package is a fork, how many imports and stars
(followers in BitBucket) for each package in the search result list.
Currently we have over 90% packages coming from these two VCS's.
Corresponding UI change is made to show these additional information
under each package's import path.
Change-Id: I669755d4b905f360918d38e8600534a61a449ba4
Reviewed-on: https://go-review.googlesource.com/24173
Reviewed-by: Alan Donovan <adonovan@google.com>
This CL moves the flags defined in database package into main and changes
them to exported variables.
Change-Id: I6c4c3287f20b506170375bbd30880d172fd39ea3
Reviewed-on: https://go-review.googlesource.com/24175
Reviewed-by: Alan Donovan <adonovan@google.com>
This change includes the new indexing functions using GAE Search API
inside indexae.go. A flag in the main package gae_search is used to
control whether to use the new search functions.
The final goal is to replace index.go with the new methods and deprecate
the current stemming algorithm and use the one from Search API.
Functions to replace original ones are implemented and labeled "2" in
their names.
Local development is unaffected with gae_search flag turned off. Local
deployment with GAE Search API needs to use dev_appserver and an
app.yaml file, which is included. The instruction on how to do so will
be added to the wiki page when the code is submitted.
Change-Id: Ia889684176bafb2d6eac075061c06a733667c914
Reviewed-on: https://go-review.googlesource.com/23794
Reviewed-by: Alan Donovan <adonovan@google.com>
Reviewed-by: Andrew Gerrand <adg@golang.org>
Add handlers for newly added Bootstrap and jQuery files. These files are
stored locally now instead of being fetched from CDN. But those static
files are not handled properly in our serve mux, causing the serve keeps
requesting them infinitely.
Change-Id: Iddc680bb889f1327ec9265fa33eff94d9fa8dec4
Reviewed-on: https://go-review.googlesource.com/23721
Reviewed-by: Andrew Gerrand <adg@golang.org>
Reviewed-by: Alan Donovan <adonovan@google.com>
Downloaded the linked version from the CDN and stored it in the
assets/ directory. Removed the Bootstrap.js Bootstra.css and jQuery.js
definition from common.html. Changed all the occurence of these
to staticPath commands in the other templates.
Fixes#410
Change-Id: Ica50da2b5c5df85d46181b77361e832128856ce9
Reviewed-on: https://go-review.googlesource.com/23335
Reviewed-by: Andrew Gerrand <adg@golang.org>
Each definition's "Uses" link points to the definition's page on
Sourcegraph, which shows where the definition is used (across all
indexed open-source repositories). This lets library users see "usage
examples" and gives library authors information about how their
library's API is being used.
It is necessary to store the value of go/doc.Func.Orig (the original
receiver name, see https://godoc.org/go/doc#Func.Orig) when building
documentation. Previously, only go/doc.Func.Recv (actual receiver
name) was stored. The original receiver name can differ from actual
due to struct embedding.
This change was originally submitted and OK'd by adg at
https://github.com/golang/gddo/pull/259.
A gddo server with this change applied is temporarily available at
http://godoc.sgdev.org/.
Change-Id: Ifa7773b56ccc08c8f063730bdf1fa441d9728d5c
Reviewed-on: https://go-review.googlesource.com/23380
Reviewed-by: Andrew Gerrand <adg@golang.org>
This fixes error when deploying to local machine.
Change-Id: I5d61c31f0c815ab3dfcaaaedf51903f7e1ec4954
Reviewed-on: https://go-review.googlesource.com/22804
Reviewed-by: Andrew Gerrand <adg@golang.org>
The current process to fetch GitHub packages check the reference first,
which is the API call I use in the change for timestamp check. It turns
out the last modified timestamp returned can be older than the most
recent commit.
This reverts commit 3abf761629.
Change-Id: I55d38f55ee118ed6069dd9b3231858cebcf95281
Reviewed-on: https://go-review.googlesource.com/22700
Reviewed-by: Andrew Gerrand <adg@golang.org>
gddo-server: pass pdoc instead of nil in the refresh call.
The package updated time is included in the header to check if it is
recently updated when requesting from VCS. Currently we use http ETag
as a general way to compare changes, which still consumes one API
request rate when requesting from Github. However, Github accepts
conditional requests using "If-Modified-Since" to check recent updates
time without counting against the rate limit if a 304 is received.
Package document is included in the refresh call to avoid unlimited
manual refreshing. The current design to allow such refresh to force
updating the package seems unnecessary.
These changes are targeting at reducing Github API request rate in order
to give us more head room when we do batch updates to do type analysis.
Change-Id: I4ec81c92662cd23dc96e5a5e141f1945b127db2c
Reviewed-on: https://go-review.googlesource.com/22506
Reviewed-by: Alan Donovan <adonovan@google.com>
Reviewed-by: Andrew Gerrand <adg@golang.org>
Log a list of package served to learn what user goes to when searching.
Log referer to learn where does user come from, such as Google search,
github reference link, etc.
Remove unnecessary handler and response writer wrapper since both are
handled by GAE request log.
This fixesgolang/gddo#390.
Change-Id: I7ac049ad7723edb229cea0dc8d0c64295981d368
Reviewed-on: https://go-review.googlesource.com/22261
Reviewed-by: Andrew Gerrand <adg@golang.org>
When running on GAE, the X-Real-Ip in header is shielded as well.
X-Appengine-User-Ip contains the real ip address in this situation.
This fixesgolang/gddo#393.
Change-Id: I64fc4aa0a16a613a4ce4727aa1425d081ce1f0c9
Reviewed-on: https://go-review.googlesource.com/22067
Reviewed-by: Alan Donovan <adonovan@google.com>
X-Real-Ip is used as the remote address to identify the original
address sent through the requests to detect robots.
This fixesgolang/gddo#393
Change-Id: I322061dd355340d864376e656185bd84fbbc9432
Reviewed-on: https://go-review.googlesource.com/21996
Reviewed-by: Andrew Gerrand <adg@golang.org>
Request to /_ah/health will get 200 response. Requests to other /_ah/
will get 404.
Change-Id: I06b809b6ee3144d4a3d0983e132b954765c95c1b
Reviewed-on: https://go-review.googlesource.com/21997
Reviewed-by: Stephen Weinberg <stephenmw@google.com>
With the move away from nginx, all redirects need to be reimplmented
in gddo-server. This implements the redirect to send talks.godoc.org
to the go-talks App Engine project and www.godoc.org to godoc.org. In
nginx these were 301 redirects. I am making them 302 redirects now
because in the future these redirects are likely to change.
go-talks is just a standard App Engine app. There is no reason we
cannot run go-talks directly from the the talks.godoc.org domain and
likely will in the future.
Change-Id: I4b3e8dda74d31daeba8bd961932f774c6ff1442f
Reviewed-on: https://go-review.googlesource.com/21588
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
With the move of godoc.org to app engine, the TLS is being stripped
before it hits the gddo-server. This causes an infinite redirect as it
continuously tries to redirect to https even though https is already
being used.
HSTS headers were also removed. Strict TLS is really not needed for
this site. It is a security feature unlikely to provide much security
while at the same time getting in the way if we ever need to host
godoc.org in a non-https enviornment.
Change-Id: Id6f7fd2e1675320ec322077c19834bf4c135ffee
Reviewed-on: https://go-review.googlesource.com/21454
Reviewed-by: Andrew Gerrand <adg@golang.org>
This change handles logging to GCE on every user request to record
user's query term, the package we serve, the status, etc. Setting a log
name in GCE to enable this logging.
Change-Id: I6d22fcd391eccc76aa9ab1d81f00a10eaba48015
Reviewed-on: https://go-review.googlesource.com/21151
Reviewed-by: Alan Donovan <adonovan@google.com>
Reviewed-by: Andrew Gerrand <adg@golang.org>
For all projects we now index the full import path. So user can search
for example gddo/doc.
Handle urls and float numbers as one term, such as v.io, 0.8.
This fixes#366, and fixes#384.
Change-Id: Ia2c95ddcb64963590edc19d5fb4d4b06ea19e018
Reviewed-on: https://go-review.googlesource.com/20563
Reviewed-by: Andrew Gerrand <adg@golang.org>
This silences all warnings except those about documentation.
Change-Id: I1d61bf871dcfd6d95b53f66d9e9378f4f648c3a0
Reviewed-on: https://go-review.googlesource.com/20464
Reviewed-by: Dmitri Shuralyov <shurcool@gmail.com>
Reviewed-by: Andrew Gerrand <adg@golang.org>
Francesc Campoy
Chris Broadfoot
Jaana Burcu Dogan
Dave Day
Tuo Shan
Dmitry Savintsev
Antonio Bibiano
Quinn Slack
Francesc Campoy
Stephen Weinberg
Dmitri Shuralyov
Andrew Gerrand
Andy Finkenstadt
ryan
Oleg Bulatov