Convert almost all of the remainder of the dochtml package to use
safehtml. (There is still a case with comments that needs to be
dealt with.)
This requires using legacy conversions to generate
safehtml.Identifiers from Go symbols and dotted expressions,
since they contain characters that are not permitted in safehtml.Identifiers.
I filed https://github.com/google/safehtml/issues/4 and
https://github.com/google/safehtml/issues/5 to get those characters
allowed.
Change-Id: Ic1b821680a6cb5dfae445a87cd305ae02cd78493
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242818
Reviewed-by: Julie Qiu <julie@golang.org>
There was an attempt to create a noscript tag, which only runs when JS
is disabled, using JS.
Remove it.
For golang/go#40321.
Change-Id: I99c02810ed7c299fb606259823ef9b764c525bb6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243858
Reviewed-by: Julie Qiu <julie@golang.org>
Modify the Renderer.declHTML method to use safehtml.
Do this by collecting safehtml.HTML fragments as the ast.Decl
is scanned, then concatenating them together.
Change-Id: Ibe45084e9e9e4015af4c4d76e21d0f255e7502c9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242497
Reviewed-by: Julie Qiu <julie@golang.org>
TestReadmeHTML/valid_markdown_readme_with_CRLF was introduced in
CL 243457 and doesn't pass. The test is fixed.
Change-Id: Ia9d12e408d9574f2cb15c3f88f82e7a309ed5b52
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243701
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Increases rollout precision to accommodate all integers in the
range [0,100].
Assigns each user a stable rollout threshold above which they are
added to and remain in the experiment, instead of being shuffled
in and out as the rollout progresses.
Fixesgolang/go#40314
Change-Id: Id1c520fe1e1659f32cc3bcc254b6729ed77309f1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243659
Reviewed-by: Julie Qiu <julie@golang.org>
Replace CRLF line terminators in readme and license files before
rendering Overview and Licenses tabs on the Details page. Keeping \r
characters interferes with blackfriday's parser, resulting in an
incorrectly rendered HTML output (see golang/go#40203 for details).
Fixesgolang/go#40203.
Change-Id: Id6c2951c9f23e7054957071cf1c33fd3fa6494c6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243457
Reviewed-by: Julie Qiu <julie@golang.org>
The quota server now checks for authentication using a known
header, to see if requests from the source should be rate-limited.
This replaces an existing implementation, where the quota server checked
for the "Referer", since this is header is usually not set by requests
that do not originate from a browser.
This functionality is used to allow the teeproxy to forward requests to
pkg.go.dev.
Change-Id: Ic1eac95c3047219842b561f2cee3b0547fe36c5c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243283
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When there are no results, the search page now provides instructions on
how to use the frontend fetch feature when that experiment is on.
For golang/go#37002
Change-Id: I28939ca0a871defb42c43bbce3dae14b657c75c1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242922
Reviewed-by: Jonathan Amsterdam <jba@google.com>
pollEvery is increased to every second instead of every 500ms, since
most fetch requests took longer than 500ms to finish when testing
locally.
For golang/go#37002
Change-Id: I6d0e77153872a6303cd60df68419adb8821dcebe
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242923
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The response text for statuses http.StatusSeeOther and
http.StatusNotFound is updated to use smart quotes and specify the
requested version.
TODOs in internal/frontend/fetch.go are also updated.
For golang/go#37002
Change-Id: I274847ca979e47dc5d9e98456d84e8511332a811
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242921
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The cacheBypassHeader header is deleted, since it isn't being used
for anything at the moment.
In the future, if we need to bypass the cache, we should use
the AuthHeader system currently used by the quota server, to determine
if a request can bypass the quota server rate-limit checks.
Change-Id: I82430df382645f93dfb433c58c5b6e014021174f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243648
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The DEFAULT isn't needed, since go_mod_path will be populated. The
syntax was also incorrect, since a column cannot both be nullable
and have a default value.
Change-Id: Idec4d1977eb070cbad13e0cc2485d7c9d804aed6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243649
Reviewed-by: Jonathan Amsterdam <jba@google.com>
version_map.go_mod_path should always be set. In events when it is
unknown (such as for status 40x requests), it will be set to the empty
string. A not-null constraint is now set on this column.
For golang/go#36811
For golang/go#37002
Change-Id: I9c5e01fbe9d772cb4cb7e43394aac2564a494734
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242437
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When we reprocess modules, ones with a status DBModuleInsertInvalid will
now be reprocessed.
Additionally, it is now possible to reprocess modules based on a
specific status code using the /reprocess endpoint.
Change-Id: I378d635ec3939717ee2c8305db7944c6467b28d3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243277
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Go symbols are used as HTML ids in span elements to allow linking to
the doc for a symbol with a fragment. Use safehtml.Identifiers for
these IDs.
Change-Id: I872b3edacf6ce053ed008dd570e92574f9189074
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242297
Reviewed-by: Julie Qiu <julie@golang.org>
When walking HTML inside markdown to translate relative image sources,
use the escaped path of the source rather than the original path.
Previously, if the value of an img src tag was escaped, like
a%20b
it was being converted to
a b
The sanitizer then removed the entire img tag.
Using URL.EscapedPath preserves the original escaping.
Change-Id: I05544a7f2d089c7026dedac605e07b193823957b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243618
Reviewed-by: Julie Qiu <julie@golang.org>
In TestReadmeHTML, move repeated ModuleInfo to a local variable.
Change-Id: Ib866715686aa4f144d21d41a7210ae65e7c5ef08
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243617
Reviewed-by: Julie Qiu <julie@golang.org>
- Walk all the children of the body node, not just the first: there
may be more than one.
- Fail if the top-level node is not what we expect, rather than
returning something.
Change-Id: I2c479064dbe45a9bbd402f3878ab4638d62dd475
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243224
Reviewed-by: Julie Qiu <julie@golang.org>
Replace the previous github template for raw data
(raw.githubusercontent...) with a new one that is more uniform (the
same as the blob template, with "raw" instead of "blob").
The main advantage is that we can drop the "repoPath" parameter to the
template, which was necessary only to support the unusual GitHub
URLs.
Now that GitHub and GitLab templates are identical, we can drop the
GitLab ones. We preserve "gitlab" in the map from kinds to templates
so we can continue to unmarshal DB modules.source_info columns
that use "gitlab".
Change-Id: Id3d45929f0eb2c040109449d4208279cca44f079
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243217
Reviewed-by: Julie Qiu <julie@golang.org>
While walking HTML, keep track of whether we made any changes.
If not, return the original HTML, not the re-rendered version.
This minimizes changes to existing readmes (since parsing HTML and
then rendering it does not always give you back the same thing).
Change-Id: Id43c1d45359d44fffd0ab9e6b655fad12262ede7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242980
Reviewed-by: Julie Qiu <julie@golang.org>
Export the ReadmeHTML function to enable testing by code outside the
package.
Specifically, we'll be writing a program to test HTML changes against
the database.
Change-Id: I21dbd2606e1c370c974e9bf65d2535860884435b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242979
Reviewed-by: Julie Qiu <julie@golang.org>
Confirm that attacks that exploit bad URLs are thwarted.
Change-Id: I079f81e0b3f6777fa6d271c4e43fccf7a0c6b438
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242978
Reviewed-by: Julie Qiu <julie@golang.org>
Change the arg to translateRelativeLink and friends to be a source.Info, since
it doesn't need a full ModuleInfo.
Change-Id: Iae92f39328c5f207c4f28ea5ba8cd4c2ca38392f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242977
Reviewed-by: Julie Qiu <julie@golang.org>
The experiment variable for executable examples was describing the
description of the variable rather than the name constant. This CL
changes it to the name constant.
For golang/go#36865
Change-Id: I84c3fcf8147c2c3964c262037ff2340f2ca0f39a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243037
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
httpCodes is renamed to codes, since several codes are not HTTP status
codes.
Change-Id: I1d0adb0e4fa26c9eb88d1fdae83fa22939de242a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242884
Reviewed-by: Jonathan Amsterdam <jba@google.com>
FromHTTPStatus is renamed to FromStatus, since several derrors codes are
not HTTP status codes.
Change-Id: I36aec502e4de2bfb900dda95a985a0fcdb3bface
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242883
Reviewed-by: Jonathan Amsterdam <jba@google.com>
ToHTTPStatus is renamed to ToStatus, since several derrors codes are
not HTTP status codes.
Change-Id: I41bf1452fdbbafe1a4f752bc3092e39515a2db4b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242882
Reviewed-by: Jonathan Amsterdam <jba@google.com>
GetLicenses is now used by the frontend to fetch license data when the
"use-directories" experiment is on.
For golang/go#39629
Change-Id: I7a4c358b9e4f0d0089c301a40703becda1d5514d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242778
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Use safehtml/template instead of html/template to render the status
page.
Change-Id: I213592acb087c2e578c9d4ed42572ea3a0949725
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242918
Reviewed-by: Julie Qiu <julie@golang.org>
This commit modifies the codeHTML function to use *doc.Example
instead of a *doc.Example.Code so that we can have access to
*doc.Example.Play whenever the example is executable. This is important so that the imports are visible in the
example documentation. If the example doesn't include a executable editor,
it will default to rendering *doc.Example.Code. Also, this commit modifies the HTML template
to include a play button for the playground integration, which will be
displayed once all of the play button functionality is incorporated.
This commit also adds an experiment flag to enable the feature by
passing in context to the render function
For golang/go#36865
Change-Id: If2ef0d128fdd791617aad5dd8eb8a08c90e9ae72
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240685
Reviewed-by: Jonathan Amsterdam <jba@google.com>
simpleImporter currently parses package names by taking the last path
component of the package path, even if the last component is the major
version. This fix ignores the version and allows simpleImporter to
return the correct name.
Fixesgolang/go#38613
Change-Id: I393db8393e5341bfe13350068f064bfe9bfc0b37
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241128
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
The teeproxy currently allows forwarding to a host specified in each
request coming from godoc.org. Use environment variables instead to
specify the target host/s for each request.
Also add a new tag to aggregate metrics by host.
Change-Id: Ifcba0f227375801f7dd154d736a84583bbbdd827
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242097
Reviewed-by: Jonathan Amsterdam <jba@google.com>
GetLicenses currently does not handle the std module or packages in the
stdlib. Support is added for those paths.
For golang/go#39629
Change-Id: I8e41cec7004111608c712b5d34f2ad07aede15b6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242777
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The functions in tab.go are rearranged so that ones with the "legacy"
prefix are moved to the bottom of the file.
For golang/go#39629
Change-Id: Ia0b1229e67eadc3dc7d26a8f4a02eab458e7de6b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242581
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The "legacy" prefix is added to fetchPackageVersionDetails and
fetchModuleVersionDetails, since these will be replaced with a
fetchVersionDetails function that reads from the paths table.
For golang/go#39629
Change-Id: I059a26cb5998b831e9be725a055ac1ce9692fee4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242580
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This change removes internal information about Google workstations from documentation.
Change-Id: I9284ee56d6fe2364b36e4146a16bfca50e7fc90a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242779
Reviewed-by: Julie Qiu <julie@golang.org>
After a badge is created when a user makes changes to the url
input field, the urls will dynamically update with JS enabled.
This change fixes the dynamically updated URLs.
Fixesgolang/go#40216
Change-Id: I6a484d3daf7e7e6b09b8a110339b505031b8fada
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242737
Reviewed-by: Julie Qiu <julie@golang.org>
This change introduces a new method on the DataSource interface,
GetLicenses, which returns the licenses that apply for a given path,
module path, and resolved version combination.
A license in the current or any parent directory of the specified path
applies to it.
Fixesgolang/go#40027
Change-Id: If91429ac12880ac3b6254bf0acd7d3ac983c93e7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241718
Reviewed-by: Julie Qiu <julie@golang.org>
Examples are collapsed by default, and if one clicks on the header, they
are expanded.
However, that's not terribly obvious to a new user, since the cursor
when hovering over those collapsed example headers is the default for a
text element, the "text selection" cursor.
Fix that, keeping the list sorted.
Change-Id: Idfb7baae1dfc3ae3b6800d58e0d95de49eabf1b7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241017
Reviewed-by: Julie Qiu <julie@golang.org>
This change fixes the urls for the markdown snippet
on the badge generation tool. The image and documentation
urls were switched.
Fixesgolang/go#40216
Change-Id: I8bfa409019f00c79116875da1ee9755b2238250e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242641
Reviewed-by: Julie Qiu <julie@golang.org>
Fixes TestFetchPathAlreadyExists/491, which was broken in CL 242458.
Change-Id: I9877a971f24874f3772dc057c978bea6d9d8bde8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242640
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Opening /http/ on pkg.go.dev is considered a shortcut to
pkg.go.dev/net/http and should redirect there. However, http.Redirect()
creates a URL relative to the current URL (/http/) and redirects to
pkg.go.dev/http/net/http instead.
In this CL, relative redirect paths are replaced by absolute paths so
that /http/ redirects to /net/http.
Fixesgolang/go#40166.
Change-Id: Ic3b9e6da59bfa2358b7f75e565b8a52662f17eb8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242138
Reviewed-by: Julie Qiu <julie@golang.org>
The canonical feedback link is changed to golang.org/s/pkgsite-feedback,
instead of golang.org/s/discovery-feedback, for consistency with the
project name. Both links will continue to work and redirect to the same
location.
Change-Id: I73fe7018c9d9935dcc584c4f90a82f6c07633787
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242557
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Status 303 is now returned for alternative modules, since 491 isn't an
actual HTTP code.
The response text for cases where we find a module but not the fullPath
is updated.
For golang/go#36811
For golang/go#37002
Change-Id: I68ca2278e4bd32c4e867d86179243419d56903bd
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242458
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Jonathan Amsterdam
Julie Qiu
Andy Bursavich
Anze Kolar
Miguel Acero
shaquilleq
Jamal Carvalho
Amarjeet Anand
Daniel Martí