The cache now performs the same checks as the quota server for a known
header to see if a request should bypass the cache. The shared header
name is now hardcoded since there's no point in keeping it in an
environment variable.
This functionality allows load-testing the frontend server without
hitting the cache.
Change-Id: I7c0fb0196ddc946e84b5fb95ea254f4ab85a3eec
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/246138
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reorganize the server so that each request gets its own DataSource,
instead of using a single DataSource for every request.
Currently, the behavior doesn't change because we do in fact use
the same DataSource for every request. But this paves the way
to having a pool of health-checked DB connections, while still
having each request work with a single connection.
For golang/go#40444.
Change-Id: I717450593a8dcfd5689a8d28f634324776305042
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244603
Reviewed-by: Julie Qiu <julie@golang.org>
This change fixes an issue in initiating the base page for a custom
error pages, which prevent fields such as the AppVersion and
GoogleTagManagerID to be set.
Change-Id: Ifdc7a058b1d6281132004036f678bebca9ef5811
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244549
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
This change creates a badge generation tool page for package authors. A future change will add
documenation of this feature and a link to the about page.
Fixesgolang/go#36982
Change-Id: Ia64ba9db73ed92b853f1f955330caf93d996da91
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241273
Reviewed-by: Andrew Bonventre <andybons@golang.org>
The Google Tag Manager ID is now set using an environment variable,
instead of hardcoding in our templates. This prevent users from sending
traffic data. to the GTM for pkg.go.dev by default, when they fork the
repository.
Change-Id: I5a1f985db20fd19378612593bde3f640b2b40d09
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241742
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This change adds a /play handler that mirrors requests
to play.golang.org/share which is used in implementing playground
integration in the frontend. This change also adds a test that
includes a mock server using net/httptest to check for proper
status codes.
Updates golang/go#36865
Change-Id: Ia3e902808534189ada747b270fed908e766c224b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241164
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The frontend now accepts POST requests, which is used for the /fetch
endpoint. The /fetch endpoint will return the standard 404 page when a
GET request is made. When a POST request is made, it will return a
plain text response.
serveDetails and serveSearch now check the request method to ensure
a GET request is being made.
Fixesgolang/go#39979.
Change-Id: Icc4d7f6ab7ebdbabdae43eab5a1c29a6446651bb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241378
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Begin using github.com/google/safehtml for frontend templates.
The safehtml module replaces html/template to provide more secure
template execution.
- Templates must come from constant strings or trusted sources.
This CL constructs TrustedSource values from the filesystem
paths that we use.
- Injected HTML must be constructed from known safe values. This CL
does this in some places. In others, notably the documentation, it
uses an escape hatch that we will close in a future CL.
- Identifiers must be constructed safely. We temporarily use an
escape hatch for license identifiers.
- There is a gotcha when using safehtml types: values that
resolve to the empty string do not evaluate to false in the
context of an `if`. We have to replace `.X` inside an `if`
with `.X.String` when `.X` is a `safehtml.HTML` or the like.
Still to be done:
- Remove all `legacyconversions.RiskilyXXX` calls (https://golang.org/issue/39960).
- Remove `template.HTML` from internal/fetch/dochtml (https://golang.org/issue/39959).
Change-Id: Icf4793d70cb2441ba30abd4e7394c4696be64a7b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240499
Reviewed-by: Julie Qiu <julie@golang.org>
The frontend fetch UI is updated with a styled button, and loading dots
to indicate that a fetch request is in progress.
The Fetch API is now used instead of AJAX to make requests.
Updates golang/go#36811
Updates golang/go#37002
Change-Id: Ia37113dd9976f8e147875371c099f7b2bfd4bd85
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240459
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Change our content security policy (CSP) for scripts.
Instead of using a nonce, which lends itself poorly to caching, use
hashes. See https://csp.withgoogle.com/docs/faq.html, search for "CSP
hashes".
To make hashes work, the hash of every inline script must appear in
our Content-Security-Policy header.
Also, not all browsers support hashing with scripts loaded from files,
so we must dynamically load the files by using an inline script that
builds a script tag with a src attribute. (We need to do this anyway
for the Google Tag Manager script.) See the link above for a
description of the technique. It works because the CSP header mentions
'strict-dynamic', which trusts everything loaded from a trusted
script.
Ideally, we would both generate all these hashes automatically,
and check that they are all correct. This CL doesn't do that.
A followup CL will.
List of changes:
- Replace script tags with scr attributes with inline scripts that
load from the files.
- In internal/middleware/secureheaders.go, add the list of script
hashes to the CSP header.
- Remove all references to nonces.
Updates b/159711607.
Change-Id: Ia9b78ecd85e24619e758f2580a370778708b9e71
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239897
Reviewed-by: Roberto Clapis <robclap8@gmail.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Disallow: /fetch/* is added to the robots.txt, so that search engines do
not crawl the endpoint and make unnecessary fetch requests.
Updates golang/go#36811
Updates golang/go#37002
Change-Id: I46d7691ba416413bdaac36ab50ed2f86dee3a7ff
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240457
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Escape any strings that may come from outside the program and will
be rendered as HTML.
Also, fix some typos.
Also, put args to a cmp.Diff call in the right order.
Change-Id: I1fc6013e200326af8719b14c28a607fc7936e20a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239478
Reviewed-by: Julie Qiu <julie@golang.org>
Remove the `cfg` global and the functions that accessed it.
Fixes b/145301722.
Change-Id: I58ab9fbd4fc29f66dbc5b120f04c88ee0703ee57
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238437
Reviewed-by: Julie Qiu <julie@golang.org>
At the moment, there are three handlers for serving the details page,
which are now combined into one. This reduces duplicated logic, and
makes it easier to support logic for requests to path@master.
Change-Id: Iedde7430ff5106ea97a4c062f9e83123fa12c5a5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/769963
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Also, fix bug where the worker change interval was being passed in.
Change-Id: Ia1fa19b231acd2d0d307ea418d1fd0d8c3a4c320
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/768541
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
When a path page 404s, it will now render the notfound.tmpl page (once
the frontend-fetch feature flag is on), which provides a button for the
user to make a request to fetch the package.
Updates golang/go#36811
Updates golang/go#37002
Updates golang/go#37106
Change-Id: I17fedd018435e8d3e51e2a2a4a972d3cf673df56
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/753606
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
A fetch endpoint is added to the frontend, which will queue a module to
be enqueued by the worker, if it doesn't already exist in the database.
After enqueuing, the fetch handler will poll the version_map table at a
constant rate, until the path returns or the request times out.
If the request fails, a corresponding statusCode and responseText will
be returned to be displayed to the user.
Updates golang/go#36811
Updates golang/go#37002
Updates golang/go#37106
Change-Id: Ic2e20146dc626bf296db05bc2abbfb50d6fd7991
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/743103
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
At the moment, taskIDChangeInterval is a hardcoded value in
internal/queue. However, we will soon have two task queues running,
which require different change intervals, so this value is now set in
internal/config.
Additionally, the taskIDChangeInterval for the worker is changed to 3
hours.
Change-Id: I498abefce6543005463be7da99a5a778f3a6e973
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/758919
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
At the moment, all errors for servePackagePageNew will return a 500,
because of derrors.Wrap. This is fixed by checking to see if the
err passed to serveError is of type serverError, and if so, return the
status code of the underlying serverError. Regardless of the error
type, the full error will be logged.
Change-Id: I502b174e1efb9f856f17ffb3614fedc909ef162c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/754806
Reviewed-by: Jonathan Amsterdam <jba@google.com>
+ Update the title suffix from go.dev to pkg.go.dev and some
incorrect usages
+ Add a meta description for SEO purposes
+ Lazy-load images in the footer (requires setting heights and
widths for both images because otherwise loading could cause
reflow
(see https://web.dev/native-lazy-loading/)
+ Use rel=noopener in places where it is not present or
where it was redundant with noreferrer. Both are not needed
(see https://web.dev/external-anchors-use-rel-noopener/)
Change-Id: I46767bd67e6db01c4d5921926381d5b32a89f018
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/752824
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
This change removes Google Analytics in favor of just Google
Tag Manager since you can use the latter to send page view data
to the former.
Additionally, adds Google Tag Manager’s debugging view CSS domain
to the content security policy.
Only renders GTM resources when not in development mode.
Change-Id: I54149c378f7226e24d836c730a8fed641d9bcb54
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/751128
Reviewed-by: Julie Qiu <julieqiu@google.com>
This change replaces the reloadTemplates field with a more general
devMode that will allow for changes to ease ergonomics when
developing locally.
Change-Id: I87132830f51070cfc3905cf739d87ddc35f2f42b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/751682
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Create test servers, their handlers, and middleware in
a single function.
Also factor out adding experiments to a context.
Change-Id: I1a4046e1d190ed7c5f3fe3d12b999bc947821b5c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/747787
Reviewed-by: Julie Qiu <julieqiu@google.com>
The frontend server is not initiated with a queue. The frontend task
queue will be used to support frontend fetches.
frontend.FetchAndUpdateState is added, which is a copy of
worker.FetchAndUpdateState for use in testing and locally.
Updates golang/go#36811
Updates golang/go#37002
Updates golang/go#37106
Change-Id: I41922d30462d2623a061aa1f207bb2b39f7b54e2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/743102
Reviewed-by: Jonathan Amsterdam <jba@google.com>
golang.org/x/discovery is renamed to golang.org/x/pkgsite.
When the repository is open sourced, it will be hosted at
go.googlesource.com/pkgsite.
Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This changes the logic in the GodocURL middleware to use the placeholder
technique used by nonce and latest to avoid caching issues interacting
badly with the display of the “Back to godoc.org” button.
The button is shown (via CSS rules) if it has a non-empty href attribute.
Updates b/147492306
Change-Id: I978e9765df2fbf887ee89f4a30ed48a955434f0c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/704879
Reviewed-by: Julie Qiu <julieqiu@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This reverts commit 0cac64d3f37dcbaa5df38ca33120b7669d381691.
Reason for revert: decided to keep all content on go.dev/about in weekly meeting
Change-Id: I901bb8951858c0af8b08da29d68a2ace044d0181
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/702040
Reviewed-by: Jonathan Amsterdam <jba@google.com>
With the option to redirect automatically from godoc.org, there needs
to be a mechanism to return to godoc.org and turn it off if necessary.
Reliably knowing that a request is coming from a godoc.org redirect
is complicated as you can’t set cross-origin cookies and 3XX redirects
don’t alter the Referer header.
In order to reliably know that a request is coming to pkg.go.dev from
godoc.org, we look for a utm_source GET parameter set to 'godoc'.
If we see this, we set a temporary cookie and redirect to the
pkg.go.dev URL with the utm_source param stripped (so that it doesn’t
remain in all our URLs coming from godoc.org). If this temporary cookie
is seen, it is marked to be deleted and the correct value for the
“Back to godoc.org” link is set. The existence of this value will be
used to determine whether to show the button in the UI.
Updates b/147492306
Change-Id: I9c2e6b891ba4f23b7490c087d32583cd7f0f6ad8
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/696565
Reviewed-by: Julie Qiu <julieqiu@google.com>
On the license-policy page, remove links to licenses
that aren't on the OSI website.
If we later find links to the other licenses, it shouldn't
be hard to add them.
Fixesgolang/go#37872.
Change-Id: Icadd9128b18c4d2b408e9726d5fec7dc9c9fe4ea
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/692259
Reviewed-by: Julie Qiu <julieqiu@google.com>
Instead of serving an error page themselves, top-level handlers return
an error with the information for serving the error page.
This is a largely mechanical rewrite that removes a lot of clumsiness
from the handling code. For example, instead of
if err != nil {
log.Errorf(ctx, "frobbing: %v", err)
s.serveErrorPage(w, r, http.StatusInternalServerError, nil)
return
}
the code is both simpler and more idiomatic:
if err != nil {
return fmt.Errorf("frobbing: %v", err)
}
It is still possible to include an errorPage with the error, like so:
if err != nil {
return &serverError{
status: http.StatusBadRequest,
epage: &errorPage{...},
err: err,
}
}
Change-Id: Id076894b1cb912fe0731fdf6fbdb7b0e5b972bcb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/686039
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Pkg.go.dev/about now redirects to go.dev/about, with status 302. This is
to make it easier to redirect godoc.org/-/about, in case we decide to
have our own about page in the future.
Change-Id: I3acfc421c99148a834b765ef9a60cd4f411631c6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/649302
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Remove the internal/license package, and all references to it.
Mostly I replace with equivalent functionality from licenses. But for
the dbpopulate command, I just remove the whole thing because it's
obsolete: it checks for NULL redistributable column but that column is
now NOT NULL.
Change-Id: I15bfe02f9f71b0adc3743b114b5d6b6dd442af6a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/644662
Reviewed-by: Julie Qiu <julieqiu@google.com>
- The requestlog middleware gets the trace ID from the request and
adds it to the request's context.
- The internal/log package retrieves the traceID and adds it to log
messages. It also creates two logs instead of one, a "parent" log
used in requestlog, and a "child" log used for all other logging.
Together, these two changes will cause the Stackdriver log viewer to
group all log messages for a request with the request start and end
log messages.
- These changes require that all log functions take a context, so I
plumbed one through everywhere. In a handful of cases it didn't seem
worth doing the plumbing. I used context.TODO() for those so we can
easily find and re-evalaute them.
Change-Id: I663588463520187d0549a8f802ba9cb44a893592
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/622940
Reviewed-by: Robert Findley <rfindley@google.com>
Now that we are on Go 1.13, we can use the errors package and remove
xerrors.
Change-Id: I8b30bcea97da25cc92b8e250299a68417a85143d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/623768
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
A fake experiment handle is used to guard the autocomplete feature
behind a flag. It defaults to false (experiment off).
Change-Id: Ib9f26c00a6e0374277c27c605aa8c9809464623f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/619084
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
frontend.Datasource is moved into internal.Datasource, so that it
can be used in future CLs in the internal/middleware package.
Change-Id: Id4823b46f36f114fd02ee3b8ac78c1e6d24c0cc5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/611020
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Previously, SearchResults was in the postgres package, but being used by
internal/frontend and internal/proxydatasource. SearchResults is now moved to
internal, so neither package has to import the internal/postgres
package.
Change-Id: I9119b6bacb07e006a9b404f6a5e71dbf483688e2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/610961
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
It's been a while since we switched to using substitution
for the latest-version badge. It's unlikely there are many
loaded browser pages with the old code.
Change-Id: I1a963963940227f10a259ca0f0b10f80d5f36850
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/611021
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Distinguish the "page title," which is put in the <title> element
and thus appears in tabs, from the title that appears at the top
of a details page.
The title in a tab should start with the resource name (e.g. "io package")
so it's readable when many tabs are open.
The page title should read more naturally (e.g. "package io").
Fixes b/145124550.
Change-Id: I29d5a92f0f51c6b2f20691dd383695dacbdd1956
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/606800
Reviewed-by: Julie Qiu <julieqiu@google.com>
This reverts commit a13a2bf99ae64c9cce01a5caaf441a9b8121afd6.
Reason for revert: tests are failing for TestServer
Change-Id: Icb3c23c158e7ea77190883e4441160535e257975
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/606421
Reviewed-by: Julie Qiu <julieqiu@google.com>
shaquilleq
Jonathan Amsterdam
Julie Qiu
Jamal Carvalho
Miguel Acero
Andrew Bonventre
Andrew Bonventre
Rob Findley