The address bar URL will be updated to be the canonical url of package.
Canonical url includes the module version in address bar.
Fixesgolang/go#36807
Change-Id: I4a6f9737ff7e112ebf1d093b2eebe2af311fb0c6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239179
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Currently the version tab displays incompatible modules first in the
list.
Example before: https://i.imgur.com/h1MjpVC.png
This change modifies the SQL queries responsible for fetching the
version list such that the list prefers non-incompatible modules,
leaving incompatible modules at the bottom of the list.
Example after: https://i.imgur.com/jbPzE7Z.png
Tests are also updated to reflect the order change.
Change-Id: I882a9ce042fce924b8e90d733aee6d717dfca7ea
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/248859
Reviewed-by: Julie Qiu <julie@golang.org>
A upperbound of 10 candidateModulePaths is added for the number of fetch
attempts that a single frontend fetch request will make. Based on our
current dataset, there aren't any module paths that currently have more
than 7 path elements.
Change-Id: Ia098e538abb5a22ba23c9d6c147ffd1da4f1fbb7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/248377
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
Errors from candidateModulePaths are now returned in
servePathNotFoundPage, which will all be bad request errors.
Previously, if a request was made to an predetermined invalid path, the
user would only get a BadRequest error after clicking "Request <path>".
The error is now returned on page load.
Change-Id: I98b25122c678ab2a59b70939cab8b74e2b627a2a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245902
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
TestProxyServer is renamed to NewClientForServer, so that the function
doesn't have a "Test" prefix when it is not a test.
SetupTestProxy is renamed to SetupTestClient, since it returns a
proxy.Client, and proxy.SetupTestClient reads better.
Change-Id: I6c3152638e29b41e61275174df5e08e7eef5b2c4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/247297
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Add a check for HTML ids that could be valid Go identifiers or dotted
expressions. Such ids are reserved for rendered documentation, which
must use them so people can jump directly to the documentation for a
symbol by typing it as a fragment.
Change-Id: I646449906a845d17ce9d3740ad13d0e0ae514b8d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242498
Reviewed-by: Julie Qiu <julie@golang.org>
The cache now performs the same checks as the quota server for a known
header to see if a request should bypass the cache. The shared header
name is now hardcoded since there's no point in keeping it in an
environment variable.
This functionality allows load-testing the frontend server without
hitting the cache.
Change-Id: I7c0fb0196ddc946e84b5fb95ea254f4ab85a3eec
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/246138
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
A metric is added to determine the number of requests that are made to
@latest, @master, or a specific semantic version. This metric is used to
get better insight into the type of requests that are being made to the
details endpoint.
The version type tag for frontend fetch is removed, since this data
isn't useful.
Change-Id: Ib89acb69297430acd3722bb805456d8ebb30730c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/246397
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When calling ScheduleFetch in serveDetails, it is possible that the
request ends and the context is canceled, before the goroutine finishes
enqueuing a task to the frontend fetch task queue.
A context.Background with a timeout is used instead.
Change-Id: Iee6909c1318c618e4ad3a275d281274c0be87355
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/246217
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
A metric is added for everytime a request is sent to play.golang.org.
The request status code will be used and is useful to have better
visibility on how many users click the new playground "play" button.
For golang/go#36865
Change-Id: I243efe5b3d8724fa6bdd60d332e2932925ffe094
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245898
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Additional source hosts are added to pre-filter invalid module paths.
Each path is also validated using module.CheckImportPath.
For golang/go#37002
Change-Id: I4090a915c023b4d84b9b139e74256c61ca183176
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245897
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This change specifies more granular page types primarily for use
by the details pages to differentiate between the type of page
(module, package, etc.) and the name of the item being viewed.
This change also hides the type of the item in the fixed header
when it is not wide enough to show other information.
Change-Id: I1e25e75dbdbac01b5d54ea96224f3464cf9fa9f9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245487
Reviewed-by: Julie Qiu <julie@golang.org>
This change implements the latest designs for the site header,
adds a new fixed header to details pages, and updates to the
latest Closure Compiler Docker image tag to support an API used
by the new fixed header implementation (ResizeObserver).
The global header is no longer sticky, but on the details page
a hybrid header that shows a combination of the header content
and details tabs is displayed when the inline tab list goes out
of view.
Both the inline and fixed tab lists are responsive in that the
elements are pushed to an overflow container when they can no
longer fit on screen. A native <select> tag is used for this to
avoid implementing a popup menu (perhaps that can be done in a
future change).
The mechanism used to copy the current path to the clipboard has
also been updated to avoid hacks that didn’t play well with
screen readers. This method does not work on iOS, but a message
is displayed to the user in that case.
Fixesgolang/go#37110
Change-Id: I2b8b73d7e85c122992b106e9b89d7e207be9f099
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245400
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
At the moment, pkg.go.dev/404 returns the following error message:
Package “404” could not be found, but you can view module “std” at
https://pkg.go.dev/mod/std.
This is because stdlib.Contains(404) returns true.
This error is fixed so that pkg.go.dev/404 just shows a normal 404
error, when frontend fetch is active.
For golang/go#37002
Change-Id: I787a10009294d105d3f92df19790dcd438123f8d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245878
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The text in fetch.tmpl is updated to provide clearer instructions.
Corresponding CSS is also adjusted for these text changes.
For golang/go#37002
Change-Id: Iae8b641eb2471172eb4465493fe83a4005bce6fc
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245880
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
The response text for different frontend fetch outcomes is
updated.
The code in fetchRequestStatusAndResponseText is also
refactored.
For golang/go#36811
For golang/go#37002
Change-Id: I175921acde5edc5cbf677eb1d1f23171b40318b9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245640
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
This change adds a boolean "replay" flag in the existing playground test for the /share endpoint. When enabled, the test will make a request to play.golang.org instead of a mock httptest server.
For golang/go#36865
Change-Id: I26bb79d21500edbc128ce46e19bed493585a3e31
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245378
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When a fetch request results in a 404, it's possible that the module
path will appear later in the index. Similarly, when a fetch request
results in a 480 or 500, it's possible that the error was transient or
has been fixed but not yet reprocessed.
In these cases, we should allow the user to refetch the path after the
taskIDChangeInterval has passed.
For golang/go#36811
For golang/go#37002
Change-Id: Ia4813140ca793b7b91d80bdc196f61af075766ca
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244601
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Previously, we used http.StatusProcessing and http.StatusNoContent to
indicate when a row did not exist in the version_map table.
Neither of these status codes fit well, so our own error code is defined
for the case when a module path has not been processed yet:
statusNotFoundInVersionMap.
For golang/go#36811
For golang/go#37002
Change-Id: Ia55972bfbfb1950ad82254d8f1082dda052bf376
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245020
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The path suffix is used in several places in the codebase. The logic for
how to get the suffix isn't necessarily intuitive, so this logic is
moved to a Suffix function.
Change-Id: Id7e6db67632d4933a8bc3ac78a7b3e0471191a6e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244437
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Previously, if the context deadline exceeded when calling GetVersionMap
in checkForPath, a 500 was returned. This is fixed so that a 408 is
returned instead.
For golang/go#36811
For golang/go#37002
Change-Id: I60d78fa0e76d5f3e4bb4dab34d66d73a45121955
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245121
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
s.fetchModule is removed. The logic from that function is moved into
checkPossibleModulePaths, which is renamed from checkPossiblePaths for
clarity.
For golang/go#36811
For golang/go#37002
Change-Id: I8ac26ac13fcddc19582682bd5351b0b7a6a2858f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245021
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
TestModule is renamed to Module, since it is shorter and proxy.Module is
equally descriptive as an exported variable.
Change-Id: Iebce664fa6e4580a4ab6b62fb73ad4bff611d067
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245118
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When recording tags for frontend fetch, the context from the request is
now used so that we know which service the request came from.
Change-Id: I2e016a734f599edac1a1db626a54026d1783b42a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245128
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Get the ExperimentSource for an Experimenter by calling a function
each time one is needed.
Part of the reorganization that will let us switch between
databases when one is bad.
Updates b/149906513.
Change-Id: I0ce6f48ad80402e9ccc3a0a3ef2cfe195f01bdc2
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244611
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Reorganize the server so that each request gets its own DataSource,
instead of using a single DataSource for every request.
Currently, the behavior doesn't change because we do in fact use
the same DataSource for every request. But this paves the way
to having a pool of health-checked DB connections, while still
having each request work with a single connection.
For golang/go#40444.
Change-Id: I717450593a8dcfd5689a8d28f634324776305042
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244603
Reviewed-by: Julie Qiu <julie@golang.org>
If a path does not exist in the paths table, but we have already
unsuccessfully fetched it in the past, display the response for that
fetch result. Otherwise, when the user click the "Fetch" button, they
will just see that response anyway.
For golang/go#37002
Change-Id: I28ac16dfd40aee15b90d68ba986862f9e1143152
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244698
Reviewed-by: Jonathan Amsterdam <jba@google.com>
frontend.FrontendFetch is repetitive, so the frontend prefix is removed.
Change-Id: Id9467ed83aff0e917984ba6ffc3ce3e5d467b151
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244697
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This change fixes an issue in initiating the base page for a custom
error pages, which prevent fields such as the AppVersion and
GoogleTagManagerID to be set.
Change-Id: Ifdc7a058b1d6281132004036f678bebca9ef5811
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244549
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Now that we have determined that walking the HTML to translate
relative image tags does no harm, use it all the time.
Change-Id: Ifcb43ed52a691daebe78029bde1f3426db7c565f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243999
Reviewed-by: Julie Qiu <julie@golang.org>
With this change, the flow of TestServer() is simplified: decision logic
whether a test should be run (based on enabled experiments) is moved to
testServer(). Required experiments can be specified when creating
individual test cases in serverTestCases().
This should make the process of adding new tests more transparent
(experiments can be allowed on the test case itself) and remove the need
for returning multiple slices of test cases (for different combinations
of experiments) which are then merged together on a per-test-server
basis. As a trade-off, additional decision logic is added to
testServer() and required experiments have to be specified on each
individual case (as opposed to a block of cases as before).
For golang/go#39663.
Change-Id: Iccdce5ddd7705e1ef8acd8aa4673d4b6a1193b2e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243017
Reviewed-by: Julie Qiu <julie@golang.org>
TestReadmeHTML/valid_markdown_readme_with_CRLF was introduced in
CL 243457 and doesn't pass. The test is fixed.
Change-Id: Ia9d12e408d9574f2cb15c3f88f82e7a309ed5b52
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243701
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Replace CRLF line terminators in readme and license files before
rendering Overview and Licenses tabs on the Details page. Keeping \r
characters interferes with blackfriday's parser, resulting in an
incorrectly rendered HTML output (see golang/go#40203 for details).
Fixesgolang/go#40203.
Change-Id: Id6c2951c9f23e7054957071cf1c33fd3fa6494c6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243457
Reviewed-by: Julie Qiu <julie@golang.org>
pollEvery is increased to every second instead of every 500ms, since
most fetch requests took longer than 500ms to finish when testing
locally.
For golang/go#37002
Change-Id: I6d0e77153872a6303cd60df68419adb8821dcebe
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242923
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The response text for statuses http.StatusSeeOther and
http.StatusNotFound is updated to use smart quotes and specify the
requested version.
TODOs in internal/frontend/fetch.go are also updated.
For golang/go#37002
Change-Id: I274847ca979e47dc5d9e98456d84e8511332a811
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242921
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When walking HTML inside markdown to translate relative image sources,
use the escaped path of the source rather than the original path.
Previously, if the value of an img src tag was escaped, like
a%20b
it was being converted to
a b
The sanitizer then removed the entire img tag.
Using URL.EscapedPath preserves the original escaping.
Change-Id: I05544a7f2d089c7026dedac605e07b193823957b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243618
Reviewed-by: Julie Qiu <julie@golang.org>
In TestReadmeHTML, move repeated ModuleInfo to a local variable.
Change-Id: Ib866715686aa4f144d21d41a7210ae65e7c5ef08
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243617
Reviewed-by: Julie Qiu <julie@golang.org>
- Walk all the children of the body node, not just the first: there
may be more than one.
- Fail if the top-level node is not what we expect, rather than
returning something.
Change-Id: I2c479064dbe45a9bbd402f3878ab4638d62dd475
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243224
Reviewed-by: Julie Qiu <julie@golang.org>
Replace the previous github template for raw data
(raw.githubusercontent...) with a new one that is more uniform (the
same as the blob template, with "raw" instead of "blob").
The main advantage is that we can drop the "repoPath" parameter to the
template, which was necessary only to support the unusual GitHub
URLs.
Now that GitHub and GitLab templates are identical, we can drop the
GitLab ones. We preserve "gitlab" in the map from kinds to templates
so we can continue to unmarshal DB modules.source_info columns
that use "gitlab".
Change-Id: Id3d45929f0eb2c040109449d4208279cca44f079
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243217
Reviewed-by: Julie Qiu <julie@golang.org>
While walking HTML, keep track of whether we made any changes.
If not, return the original HTML, not the re-rendered version.
This minimizes changes to existing readmes (since parsing HTML and
then rendering it does not always give you back the same thing).
Change-Id: Id43c1d45359d44fffd0ab9e6b655fad12262ede7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242980
Reviewed-by: Julie Qiu <julie@golang.org>
Export the ReadmeHTML function to enable testing by code outside the
package.
Specifically, we'll be writing a program to test HTML changes against
the database.
Change-Id: I21dbd2606e1c370c974e9bf65d2535860884435b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242979
Reviewed-by: Julie Qiu <julie@golang.org>
Confirm that attacks that exploit bad URLs are thwarted.
Change-Id: I079f81e0b3f6777fa6d271c4e43fccf7a0c6b438
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242978
Reviewed-by: Julie Qiu <julie@golang.org>
Change the arg to translateRelativeLink and friends to be a source.Info, since
it doesn't need a full ModuleInfo.
Change-Id: Iae92f39328c5f207c4f28ea5ba8cd4c2ca38392f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242977
Reviewed-by: Julie Qiu <julie@golang.org>
ToHTTPStatus is renamed to ToStatus, since several derrors codes are
not HTTP status codes.
Change-Id: I41bf1452fdbbafe1a4f752bc3092e39515a2db4b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242882
Reviewed-by: Jonathan Amsterdam <jba@google.com>
GetLicenses is now used by the frontend to fetch license data when the
"use-directories" experiment is on.
For golang/go#39629
Change-Id: I7a4c358b9e4f0d0089c301a40703becda1d5514d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242778
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Kush Patel
Miguel Acero
Julie Qiu
Jonathan Amsterdam
shaquilleq
Andrew Bonventre
Anze Kolar
