When the URI path length exceeds 1000, a Status 414
(StatusRequestURITooLong) is now returned. There arent any valid paths
on pkg.go.dev that will require that long of a URL length.
AcceptMethods is renamed to AcceptRequests, since it now also performs
this URI check.
Change-Id: I5fc88a57ef3a78169721d0e53a6d1d7f0f46de3f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/248378
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
TestProxyServer is renamed to NewClientForServer, so that the function
doesn't have a "Test" prefix when it is not a test.
SetupTestProxy is renamed to SetupTestClient, since it returns a
proxy.Client, and proxy.SetupTestClient reads better.
Change-Id: I6c3152638e29b41e61275174df5e08e7eef5b2c4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/247297
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Add a mode to postgres.DB where data is saved even if the module or
package is not redistributable. This "bypassing license restrictions"
mode is off by default.
Add a flag to the worker binary to turn it on. Used a flag rather than
an environment variable so there is no way for ambient state to affect
the binary for this important decision.
Also:
- Add a test to verify that license data is removed/not removed
depending on the bypass setting.
- Fix a bug where the new model insertion wasn't omitting
documentation for non-redistributable modules.
- Fix a bug in internal/sample where the top-level readme wasn't
getting populated.
A later CL will change the frontend to bypass the check.
For golang/go#39602
For golang/go#39629
Change-Id: I67a6d24c18f3b93cfbfc9ec2a20159c07a84e077
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/246957
Reviewed-by: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
The cache now performs the same checks as the quota server for a known
header to see if a request should bypass the cache. The shared header
name is now hardcoded since there's no point in keeping it in an
environment variable.
This functionality allows load-testing the frontend server without
hitting the cache.
Change-Id: I7c0fb0196ddc946e84b5fb95ea254f4ab85a3eec
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/246138
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This change specifies more granular page types primarily for use
by the details pages to differentiate between the type of page
(module, package, etc.) and the name of the item being viewed.
This change also hides the type of the item in the fixed header
when it is not wide enough to show other information.
Change-Id: I1e25e75dbdbac01b5d54ea96224f3464cf9fa9f9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245487
Reviewed-by: Julie Qiu <julie@golang.org>
Pass TEMPLATE=template0, LC_COLLATE='C' and LC_CTYPE='C' to CREATE
DATABASE statements. If not explicitly set, PostgreSQL will use
template1 by default for which we cannot control collations and ctypes.
dbtools.CreateDBIfNotExists() and postgres.recreateDB() are also broken
up into smaller functions performing a single operation on the database
to remove duplicated code.
Fixesgolang/go#40347
Change-Id: I66130ce5b56c95f647cccedf1f3d1358db33dd24
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244517
Reviewed-by: Julie Qiu <julie@golang.org>
TestModule is renamed to Module, since it is shorter and proxy.Module is
equally descriptive as an exported variable.
Change-Id: Iebce664fa6e4580a4ab6b62fb73ad4bff611d067
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245118
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Get the ExperimentSource for an Experimenter by calling a function
each time one is needed.
Part of the reorganization that will let us switch between
databases when one is bad.
Updates b/149906513.
Change-Id: I0ce6f48ad80402e9ccc3a0a3ef2cfe195f01bdc2
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244611
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Reorganize the server so that each request gets its own DataSource,
instead of using a single DataSource for every request.
Currently, the behavior doesn't change because we do in fact use
the same DataSource for every request. But this paves the way
to having a pool of health-checked DB connections, while still
having each request work with a single connection.
For golang/go#40444.
Change-Id: I717450593a8dcfd5689a8d28f634324776305042
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/244603
Reviewed-by: Julie Qiu <julie@golang.org>
Use safehtml/template instead of html/template to render the status
page.
Change-Id: I213592acb087c2e578c9d4ed42572ea3a0949725
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242918
Reviewed-by: Julie Qiu <julie@golang.org>
The New suffix is removed from PackageNew and DirectoryNew, since the
legacy methods are already prefixed with "Legacy".
For golang/go#39629
Change-Id: I8be32310c6b451035021663000aac6ca845ee6e0
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241900
Reviewed-by: Jonathan Amsterdam <jba@google.com>
GetPackagesInDirectory is added, which uses the paths table to get all
of the packages within a given directory path. This will replace
LegacyGetDirectory and LegacyGetPackagesInModule in creating the
directories tab.
For golang/go#39629
Change-Id: I4b39690e838756421bee9b5ca389081e184761bf
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241838
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The frontend now accepts POST requests, which is used for the /fetch
endpoint. The /fetch endpoint will return the standard 404 page when a
GET request is made. When a POST request is made, it will return a
plain text response.
serveDetails and serveSearch now check the request method to ensure
a GET request is being made.
Fixesgolang/go#39979.
Change-Id: Icc4d7f6ab7ebdbabdae43eab5a1c29a6446651bb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241378
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The insert-directories experiment flag is deleted, since we have already
inserted data for all modules into the paths, package_imports,
documentation, and readmes table, and been running that code path for a
while.
Updates golang/go#39629
Change-Id: I323850a462672c41ad0c67b6ab2b173bb32bf441
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241320
Reviewed-by: Jonathan Amsterdam <jba@google.com>
An integration test is added for fetching and viewing the master version
of a package and module.
Updates golang/go#36811
Change-Id: I30452e248d1ff8cdb0ef21779fcd06aafd8f0250
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240946
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Change the type of the documentation fields from string to safehtml.HTML.
This forces the conversions between strings and safehtml.HTML values
to happen at the lowest level, reading and writing to the database.
That increases our confidence that nothing will modify the HTML after
it is read from the DB and before it is used in a template.
Change-Id: If5d65123ce2b69ef3183221048aa6d081b003762
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240510
Reviewed-by: Julie Qiu <julie@golang.org>
Create anchors for license links that are guaranteed safe.
We can't use file paths alone, because they may contain invalid
characters. It's simplest to use the position of the license in
the list, sorted by path.
Change-Id: I069fe9d15309943cd2007c817882171543387929
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240501
Reviewed-by: Julie Qiu <julie@golang.org>
Begin using github.com/google/safehtml for frontend templates.
The safehtml module replaces html/template to provide more secure
template execution.
- Templates must come from constant strings or trusted sources.
This CL constructs TrustedSource values from the filesystem
paths that we use.
- Injected HTML must be constructed from known safe values. This CL
does this in some places. In others, notably the documentation, it
uses an escape hatch that we will close in a future CL.
- Identifiers must be constructed safely. We temporarily use an
escape hatch for license identifiers.
- There is a gotcha when using safehtml types: values that
resolve to the empty string do not evaluate to false in the
context of an `if`. We have to replace `.X` inside an `if`
with `.X.String` when `.X` is a `safehtml.HTML` or the like.
Still to be done:
- Remove all `legacyconversions.RiskilyXXX` calls (https://golang.org/issue/39960).
- Remove `template.HTML` from internal/fetch/dochtml (https://golang.org/issue/39959).
Change-Id: Icf4793d70cb2441ba30abd4e7394c4696be64a7b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240499
Reviewed-by: Julie Qiu <julie@golang.org>
GitHub paths typically have three components. Change our sample module
path to match that URL pattern.
Change-Id: I049df016b46affd942209a2aee798fdb5537e0a6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240943
Reviewed-by: Jonathan Amsterdam <jba@google.com>
There has been progress made since the package comments
were written. Update them to reflect the current state.
For golang/go#39621.
Change-Id: I3f950428a96bc0561b45b0d49117d2eda58d3cf7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240187
Reviewed-by: Julie Qiu <julie@golang.org>
Instead of generating the entire breadcrumb HTML in code, put the
static bits in details.tmpl and build the dynamic data in code.
This is not only easier to read and maintain, but it also removes
some uses of template.HTML, which we're trying to eliminate.
Change-Id: Ic2868b88d15496bbe4bccff6abb82481e7358b7d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240437
Reviewed-by: Julie Qiu <julie@golang.org>
postgres.DeleteModule is updated to delete the corresponding module_path
and resolved_version row from the version_map table.
Fixes#39633
Change-Id: I9de46b08e535bea52c2ea4dbab1c71a8e1c2c2f7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238637
Reviewed-by: Julie Qiu <julie@golang.org>
InMemory now expects the given function to close over things like the
proxy and source clients.
Change-Id: I7b3a2793a824ca29453b19b47b96bdedb2a91010
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238441
Reviewed-by: Julie Qiu <julie@golang.org>
Add a /poll endpoint that polls the index, but just writes
to module_version_states without enqueuing.
Add an /enqueue endpoint that is identical to /requeue. The name more
accurately reflects that it is for new modules as well as
reprocessing.
Update the worker status page to use the new endpoints.
We'll delete the old endpoints after deploying and changing the
scheduler jobs.
Updates b/158866584.
Change-Id: Id116bf9fd99fa55aaacd71bb4ca6b60770ca8812
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239480
Reviewed-by: Julie Qiu <julie@golang.org>
Remove the `cfg` global and the functions that accessed it.
Fixes b/145301722.
Change-Id: I58ab9fbd4fc29f66dbc5b120f04c88ee0703ee57
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238437
Reviewed-by: Julie Qiu <julie@golang.org>
This is the first CL to clean up various TODOs that are outdated, and
replace internal issue links with GitHub issue links.
Updates golang/go#39621
Change-Id: If270e8b2e8198c007cb4aa71ad8486182f4f3380
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238319
Reviewed-by: Jonathan Amsterdam <jba@google.com>
DirectoryMeta is added, which contains metadata for a directory.
This struct will be used to return the list of directories in a given
path in a follow up CL.
Change-Id: I89880bd8104791510bc07c1fb3532f6a71bed24c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238245
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Also, fix bug where the worker change interval was being passed in.
Change-Id: Ia1fa19b231acd2d0d307ea418d1fd0d8c3a4c320
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/768541
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
It was hard to read the args to NewServer, and now that we have two
redis clients, it was error-prone to provide them (order matters).
Using a struct effectively lets the caller name the args.
Change-Id: I0e2e39e09402031fd21a754961a2685c377c75fc
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/768540
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
We sometimes want to clear our Redis-based cache. Adding an endpoint
to the worker will let us do this more simply and with less chance of
error than running the redis CLI.
Change-Id: I855ea5d906f0cb080b4e2f6d5fa279a6a2e0b949
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/768539
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
ModuleInfo is added, which represents the module info for the new data
model. ModuleInfo is embedded in LegacyModuleInfo.
In order to support the existing overview tab functionality,
GetDirectoryNew was changed to return the module README, regardless of
whether there is a README for the directory.
We will change this logic to display the README for the directory in a
future CL.
Change-Id: I624a6d99b711870826fd7dff9100d4ad47852db2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766801
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When we insert module information, make sure we overwrite existing
rows. Previously, we specified that new data should be ignored if the
row already existed, which meant that we were adding rows but never
changing them. That didn't matter at one point, when we deleted a
module before re-inserting it, but we no longer delete, so we must
upsert.
We also fix a number of places where our test modules had duplicate
directories. The upserts failed for these.
Change-Id: I97465b11e4ea6cbb7835e883f36f098445207eba
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766365
Reviewed-by: Julie Qiu <julieqiu@google.com>
ModuleInfo is renamed to LegacyModuleInfo, as a step towards
deprecating LegacyReadmeFilePath and LegacyReadmeContents.
In a follow up CL, we will add ModuleInfo as an embedded struct
to LegacyModuleInfo.
Change-Id: Ie452420448eec1d13edaf62e548df0b9e2cbbe4b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766479
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The following structs have been renamed with a Legacy prefix:
* internal.Directory
* internal.Package
* internal.VersionedPackage
* sample.Package
The following fields on internal.ModuleInfo have also been changed:
* ReadmeFilePath
* ReadmeFileContents
This is done to help us distinguish between legacy and method
structs/methods while migrating code to the new data model.
Change-Id: Ibedf71d4db6323ef5aa05d73a0240537ea6073d3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/765160
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Rather than reading experiments from the context, the experiment set for
an InMemoryQueue is now pass in as an argument. This makes it more
explicit what experiments are being set.
Change-Id: Ib68f567ea5b7ff0fc2157ad2713c76d33827c442
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/759927
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
At the moment, taskIDChangeInterval is a hardcoded value in
internal/queue. However, we will soon have two task queues running,
which require different change intervals, so this value is now set in
internal/config.
Additionally, the taskIDChangeInterval for the worker is changed to 3
hours.
Change-Id: I498abefce6543005463be7da99a5a778f3a6e973
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/758919
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Construct the sample SourceInfo from the given module path, not
the global one.
Change-Id: I2d61d0b234c591a2a6fb03bfbf329549cc0df507
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750122
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
The frontend server is not initiated with a queue. The frontend task
queue will be used to support frontend fetches.
frontend.FetchAndUpdateState is added, which is a copy of
worker.FetchAndUpdateState for use in testing and locally.
Updates golang/go#36811
Updates golang/go#37002
Updates golang/go#37106
Change-Id: I41922d30462d2623a061aa1f207bb2b39f7b54e2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/743102
Reviewed-by: Jonathan Amsterdam <jba@google.com>
FetchModule now only returns FetchResult, which has been refactored to
contain any errors that occurred.
This allows us to skip populating various fields in fetchTasks.
Change-Id: I99bf5ce1f10461f42da9f62c77f0e821af926323
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/743101
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Use GetPathInfo to determine how to serve a path.
Protect behind an experiment.
Change-Id: Ia78b0049e4d76216654216c66a6c0786f38bd8f5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/731801
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Replace many calls to sample.DefaultModule, and all calls to
sample.DefaultPackage, with calls to the new Module and Package
functions.
Change-Id: I76921e14502585f8ca9a4dba5de01d7055522f3f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/733604
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Redo the arguments to Package so it can derive the Name
and V1Path fields from them.
- Add a list of suffixes to the arguments of Module.
Change-Id: I966197846b49befceb04779c9a04162957fbeb87
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/733603
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
This is the first step towards making sample clearer and less
error-prone.
- Rename Module and Package to DefaultModule and DefaultPackage, so
existing code continues to work. In later CLs, I'll replace most or
all calls to these functions.
- Define ModuleInfo, Module and Package to take as arguments the most
common fields. The Module function does _not_ create a package; it
returns a module with no packages.
- Most important: define AddPackage, which adds a Package to a Module
and correctly updates Module.Directories.
- Remove VersionedPackage, since it's clearer to create one directly.
- Update a couple of call sites, just to get tests to pass. More
updates in a later CL.
Change-Id: I46eb94ba897d4f122483b58435107b8782c6044f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/733619
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
golang.org/x/discovery is renamed to golang.org/x/pkgsite.
When the repository is open sourced, it will be hosted at
go.googlesource.com/pkgsite.
Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273
Reviewed-by: Jonathan Amsterdam <jba@google.com>
sourceClient wasn't being initialized by worker.NewServer, which led to
a nil pointer error that is now fixed.
Change-Id: I5d16e69e60c9a1ba489a0b5ec0e3a1a6a1f19366
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/711165
Reviewed-by: Jonathan Amsterdam <jba@google.com>
proxy.TestVersion is deprecated in favor of proxy.TestModule.
TestModule represents a module version used to generate testdata.
Change-Id: I718045917a352af218ccebb45fa0d2875a5b9610
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/709704
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Julie Qiu
Jonathan Amsterdam
shaquilleq
Andrew Bonventre
Anze Kolar
Dmitri Shuralyov
Bharath Thiruveedula