With the option to redirect automatically from godoc.org, there needs
to be a mechanism to return to godoc.org and turn it off if necessary.
Reliably knowing that a request is coming from a godoc.org redirect
is complicated as you can’t set cross-origin cookies and 3XX redirects
don’t alter the Referer header.
In order to reliably know that a request is coming to pkg.go.dev from
godoc.org, we look for a utm_source GET parameter set to 'godoc'.
If we see this, we set a temporary cookie and redirect to the
pkg.go.dev URL with the utm_source param stripped (so that it doesn’t
remain in all our URLs coming from godoc.org). If this temporary cookie
is seen, it is marked to be deleted and the correct value for the
“Back to godoc.org” link is set. The existence of this value will be
used to determine whether to show the button in the UI.
Updates b/147492306
Change-Id: I9c2e6b891ba4f23b7490c087d32583cd7f0f6ad8
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/696565
Reviewed-by: Julie Qiu <julieqiu@google.com>
This CL only renames directories, and makes the minimal
changes needed for tests to pass.
Updates b/150864416.
Change-Id: Iebe977a1a80341e170c306e58851fa1b48ef79e6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/686700
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
This keeps them under version control, but makes it easier to
remove them from the repo.
Updates b/150142069.
Change-Id: I7811f31cc35a2dd8125e43a3876115d2c6f18ad2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/682844
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
I ran unparam (`go get mvdan.cc/unparam`) on our codebase, and
it found a few issues that I fixed.
- etl/main.go:getRedis: Use the context to set the DialTimout option when dialing redis.
- config.go:readOverrideFile: Use the bucketName argument, not a global constant.
- various: Remove unused parameters and return values.
Change-Id: I98a1e5d728d41033b5f632ccc6c22442d91c2b6d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/678587
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Provide a way to configure a secondary DB host.
- Modify the frontend to try connecting to the secondary if the primary fails.
This is a partial solution to the problem discussed in the bug below. It deals
with DBs that are bad on startup, but not ones that go bad later. For that case,
we have to wait for the process to die, or kill it manually, so on restart it
can pick the secondary DB.
Updates b/149906513.
Change-Id: I177590111dc4544476a4b2aef39f06456ba28ab5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/673861
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
As per the bug listed below, we are trying to remove the global cfg
variable from the config package. This CL brings us close to that
goal. There are still two uses of the global that are harder to remove.
Updates b/145301722.
Change-Id: Ia6c350ef5d817ff7fb3125cd207343d10911d64f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/672580
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
The code for creating a new package is moved to internal/queue. This
allows for adding items to a task queue from the frontend, without
importing the entire internal/etl package.
No logic is changed, except the signature of
queue.InMemoryQueue.process, so that etl.FetchAndUpdateState can be
passed in, and the entire internal/etl package does not need to be
imported by internal/queue, which would lead to a circular import.
Updates b/135954292
Change-Id: I33a6418c73d85e15c4aa5593d633e19eb7e4eb1b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/671014
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The author changed their mind and wants unexcluded
github.com/clevergo/clevergo back on the site.
Change-Id: I07a6c25dda1fb2861ca9553195cd4707f309e3a9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/664075
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Permanent exclusions (user requests, malicious license violations, etc.) are
at risk of vanishing if the DB is corrupted or truncated. Keep the
list in code (it's pretty small), and insert it on etl startup.
We clumsily check if it's present, then insert if it isn't. The alternatives
are: (1) check the error to see if it is a primary key violation, but that
requires looking into the pq package's errors, and we don't want to do that
(what if we switch DB clients?); or (2) change InsertedExcludedPrefix to
do some other conflict resolution, but that wouldn't be right for its
primary use with dbadmin.
Change-Id: If346d97326b3ef566eb0c9ce9387c05645e47955
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/663154
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
In preparation for eliminating the cfg singleton, export the Config type
and add a load function that resolves a Config from the environment.
The Init method is kept around temporarily, but modified to return the
Config that was loaded. This allows for incrementally deprecating
dependence on the config singleton. In the future, the Init function
will be deprecated and Load will be exported, making the package
stateless.
Configuration values that were only used in main.go are now read
directly from the newly available Config instance. This allowed deleting
some functions in the config package.
Updates b/145301722
Change-Id: I98e00971fad95cc4d69b371d815de5f5498b04db
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/609591
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
An experiments middleware is added, which is used in the frontend
binary.
The experiments middleware periodically polls the provided experiments
source for information, and decides whether a given request should be
enrolled for every experiment in the set.
Updates b/146052411
Change-Id: I1ee75cb20fe1d6f5cca1a7ad0995e2147e9820f3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/620409
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
FetchAndInsertVersion is moved back to the internal/etl, after it was
initally moved to internal/fetch in commit
4252193cf5471489b9cb2c4351427448ac422e26, because it is only used by
the internal/etl package.
Fixes b/146890756
Change-Id: I4945f07636dc8986ad80f36f916c2676f85e2ab7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/627781
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
We currently import internal/etl into internal/proxydatasource,
because etl.FetchVersion is used by the proxydatasource.
These functions are moved into their own package fetch.
FetchAndInsertVersion may be moved back to internal/etl in a later CL,
since it is only used by the ETL. At the moment, all functionality for
FetchVersion is being tested by FetchAndInsertVersion.
Updates b/146890756
Change-Id: Icaaf1feeac3821c82a85f21e07085d65556328d5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/626675
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
- The requestlog middleware gets the trace ID from the request and
adds it to the request's context.
- The internal/log package retrieves the traceID and adds it to log
messages. It also creates two logs instead of one, a "parent" log
used in requestlog, and a "child" log used for all other logging.
Together, these two changes will cause the Stackdriver log viewer to
group all log messages for a request with the request start and end
log messages.
- These changes require that all log functions take a context, so I
plumbed one through everywhere. In a handful of cases it didn't seem
worth doing the plumbing. I used context.TODO() for those so we can
easily find and re-evalaute them.
Change-Id: I663588463520187d0549a8f802ba9cb44a893592
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/622940
Reviewed-by: Robert Findley <rfindley@google.com>
frontend.Datasource is moved into internal.Datasource, so that it
can be used in future CLs in the internal/middleware package.
Change-Id: Id4823b46f36f114fd02ee3b8ac78c1e6d24c0cc5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/611020
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Make the middleware timeout shorter, to distinguish it from
a possible nginx timeout.
Might help in understanding a repeat of
Change-Id: Iee2628ef19c827586e42bd0accd247ac6c2246ec
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/608334
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
The constructor for a postgres.DB takes a *database.DB
instead of a driver and connection string.
Change-Id: Ief58ce676946fd34cbcafc2b634a772663b6f932
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/604403
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Now that completion data is being populated by the ETL, we can use it to
provide a JSON API for search autocompletions at /autocomplete?q=query.
This API reads lexical completions from Redis, sorts according to a
scoring heuristic, and returns the results as a JSON array.
For testing purposes, the pathCompletions function is moved from the etl
package to the complete package, and exported.
Updates b/143370178
Change-Id: I61d75296399222fa9ebfe8aadc909bf048389a23
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602064
Reviewed-by: Jonathan Amsterdam <jba@google.com>
We're going to compute auto-completions using redis sorted sets in the
'HA' redis instance (because we don't want them to be evicted by
caching). This data need not be very fresh, and so can be computed by
the ETL every so often, perhaps once a day.
This CL wires the HA redis instance into the ETL, and adds an action to
update these Redis sorted sets. Doing this takes 1-2 minutes and
consumes quite a lot of memory, so there are some optimizations used:
+ the underlying *sql.DB is used for streaming search documents, rather
than a method on postgres.DB. There is explanation for this in the
code, and additionally I don't think this is so bad as long as it is
only done in the ETL.
+ Completions data is loaded into temporary Redis keys and then renamed
to overwrite the keys that will be used by the frontend, in order to
minimize disruption while the completion data is being updated.
Updates b/143370178
Change-Id: I48caf77eb26d8ea7b60c2e117481bfa98ef089ca
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602062
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Use server-side string replacement to determine what to show for the
latest-version badge, rather than client-side code that calls back to
the server.
Using the same technique as for script nonces, we embed placeholders
in the HTML, and a middleware layer replaces them with the correct
values. The HTML with placeholders is indefinitely cacheable,
yet we still serve a page whose body can change dynamically.
The replaced information is kept to a minimum: the latest version for
use in the "go to latest" link, and a CSS class. Both alternatives for
the badge are in the HTML, and the CSS class determines which one is
displayed.
An additional challenge we did not face with nonces: how to extract
information embedded on the page that we must use to determine the
placeholder values. In this case, that is the version being served,
the module path, and in the case of a package, the package path. We
could find that information by parsing the HTML, but that is overkill,
and slow.
I first tried embedding the information in an HTML comment and
used a regexp to extract it, but surprisingly, HTML comments are
stripped out by html/template, and there is no easy workaround. So I
added the information to the HTML as data-XXX attributes, but I still
extract them by regexp for speed. Since this data appears before any
user-provided text (like documentation or a README), and we find the
first match, there is no way for users to spoof it. But it is fragile,
as the comment in details.tmpl points out.
Now that the latest-version badge is done on the server, we can
add checks for it to TestServer, which I've done.
Fixes b/144098197.
Change-Id: I3a126f1cac48cd35c0317d47035bf227483cdbf1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599082
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
A new middleware is added that sends an error report whenever the ETL
returns a 500 from a scheduled endpoint.
Along the way, refactor main() in cmd/etl to be more consistent about
resolving dependencies in helper functions.
Fixes b/142731446
Change-Id: Ib44cd40bf48a3c229220c8c571e52ee033047aa5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/592534
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Simply exporting the distribution is not enough: in order to alert on a
number or ratio of failed probes, we need to have a separate view that
aggregates by count.
Updates b/143915414
Change-Id: I73f7239d41a65a608d3afebb408c3f54b30a8af6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/592544
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
The stackdriver exporter prepends custom.googleapis.com/Opencensus to
metrics, which means the custom.googleapis.com I added was redundant.
..
All alert policies and dashboards will also have to be updated, which
I'll do in a later CL.
Updates b/142737661
Change-Id: I25a5035ad01add59d4b34bcd0a1851bd87a6c35e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/592541
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Right now, if any of our frontend handlers panic the user will be served
a blank page. This CL adds a new middleware to recover from panics and
serve a provided panic handler.
This is then hooked into the frontend, using our default error page.
Fixes b/141254586
Change-Id: Ie45b0f5c83ff88fc5bec2ab7e174fa0ea2bd0253
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/589272
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Make it possible to reconfigure App Engine instances dynamically, without
redeploying them.
With this feature, some config settings can be changed just by copying
a file to GCS. Instances will get the new config when they restart.
- middleware/quota.go: move QuotaSettings to config to avoid circular import
- config.Init: read and decode the config overrides from a file in GCS
- Add the env var for the config override object to config/{dev,staging,prod}.yaml.
- Add sample files config/{dev,staging,prod}-override.yaml.
Change-Id: I9b13d02a5b50748b45172202c5261330b2f9f81d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/589173
Reviewed-by: Julie Qiu <julieqiu@google.com>
Now that we're caching, our probes are no longer exercising the
database. This CL configures several probes that bypass the cache.
Change-Id: Ia87239ab4ca34ace5f384880b45bd01a108bcf31
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/584257
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add the Quota middleware to the chain, but configure it only
to collect metrics about how many requests it would have blocked;
do not actually block anything.
Change-Id: I6ed6fbc1688fd35211e090507004cc5efb2fc90d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/583655
Reviewed-by: Robert Findley <rfindley@google.com>
To make pages cacheable, we cannot embed the actual nonce value in the
cached body, because it must be different on each request. So we instead
generate the body with a placeholder, and substitute in the actual
nonce value when we serve it.
Change-Id: I3f8397fbc594ee9081b3426dc5b4048d3694053a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/582017
Reviewed-by: Robert Findley <rfindley@google.com>
Optionally read a file of module@version that were removed
by the proxy, but still in the index. Exclude these from
being fetched.
Change-Id: I26269e0aba29a09fa86fcac462b4ade0ee7947d0
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/580836
Reviewed-by: Julie Qiu <julieqiu@google.com>
A page caching middleware is added that caches any 200 OK responses for
a configurable duration. It is instrumented with custom metrics.
For now, this isn't wired into the frontend, though the code to do so is
commented out. Tests will be added in a follow-up CL.
Updates 143163521
Change-Id: Id919d3e63e1b43f5905e7f20df475a3c69379e87
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/580776
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
A new method is added to postgres.DB, optimistically named 'FastSearch',
that implements search hedging. Each FastSearch concurrently queries
both all packages as well as popular subsets of packages, so that
results can be returned quicker when they consist entirely of popular
packages.
Additionallly SearchResult.Rank is renamed to 'Score', to differentiate
it from ts_rank.
The old 'Search' method is kept for now, although 'log' is corrected to
'ln' (this should have little effect on the ordering of results). In a
subsequent CL we will switch the default search method to FastSearch,
and implement the 'of many' qualifier in the search result summary.
Updates b/141182438
Change-Id: I73c6700e09288f3a8bfdb5fa57d0899606e452d2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/558516
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The following URLs are added to the prober:
* /pkg/cloud.google.com/go/firestore?tab=<tab> for readme and imports
* /search?q=github
* /pkg/golang.org/x/tools/go (directory page)
* /search-help
* /license-policy
Change-Id: I3f0836adbb46d68e8a82cfc5be95245ec5b46f87
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/558321
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Take advantage of log levels when logging DB queries: use
Debug, except for errors.
Change-Id: Icd99d1474799b89d2d8149e5f5bb6f417c6448b5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/556725
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Use the new logger everywhere.
Also remove the Printf alias; it is easy enough to change all call sites.
Change-Id: I4f46c306e22f132cd505518a8d46b1a989f6bd1d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/556722
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
It is intended as a drop-in replacement for the standard log
package, but also supports levels (just three for now)
and structured logging.
Also, converted etl/main.go as an example, to show usage.
If this is approved I'll convert everything else.
Change-Id: I8ee8855be05aafdd31cbf9b550aae76cb2240abe
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/556096
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Our usage of opencensus views is updated so that we only export 'custom
metrics' to StackDriver that start with the custom.googleapis.com
prefix. Additionally views are updated to always contain the server
route tag or client host tag.
Fixes b/141183559
Change-Id: Ie17db025bd61a3cc1ea969d214947d09cdacf96a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/553907
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Andrew Bonventre
Jonathan Amsterdam
Julie Qiu
Rob Findley