To avoid repetition, the "Version" suffix in version.LatestVersion,
version.MainVersion, and version.MasterVersion are dropped.
The current version.Latest function is renamed to version.LatestVersion.
Change-Id: I649229939223b686dd95a137df4871a899f48aab
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/325390
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Because the constants for "latest", "master" and "main" were in package
internal, we couldn't use them in internal/stdlib due to an import
cycle.
They are now moved to internal/version to avoid this cycle.
Change-Id: I6ebe7c0f50037be2b047c4a25054a0531dcfd3f8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/325389
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
We weren't reporting on NotFound, but the right code is NotFetched,
since that is what is returned when proxy fetch is disabled.
Change-Id: I554d5186fb4db0346fb9456fe9fc55091066abda
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320751
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Previously on the versions page, we were not handling the case when
different identifiers are added for the same type for different build
contexts, and which build context was surfaced was based on chance.
To support this case, it is now possible to show multiple of the same
type at the same version. For example,
https://pkg.go.dev/internal/poll?tab=versions at go1.10 will show:
```
type FD — windows/amd64
+ func (fd *FD) ReadMsg(p []byte, oob []byte) (int, int, int, syscall.Sockaddr, error)
+ func (fd *FD) WriteMsg(p []byte, oob []byte, sa syscall.Sockaddr) (int, int, error)
type FD — darwin/amd64, linux/amd64
+ func (fd *FD) SetBlocking() error
+ func (fd *FD) WriteOnce(p []byte) (int, error)
```
For golang/go#37102
Change-Id: I19e6ef12f1f8f9c412aab7cea2782409eecf29f9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/317489
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
An integration test is added for these cases:
- A symbol changes across different build contexts
- A symbol is introduced at different versions for different build contexts
- The package symbol for a symbol changes because its parent changes,
but the name and synopsis are the same
For golang/go#37102
Change-Id: Ieca17042e11b2fd583e2133a3064ab6043a29858
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/316969
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The frontend now displays different synopsis for the same symbol, if
they change based on build context.
For golang/go#37102
Change-Id: Id0887efbcc263434e5d25b9149e480c005d09f2d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/316549
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Rather than generating a module path that always begins with
"example.com", readTxtarModule now reads the module path from the go.mod
section of the txtar file when one is present.
Change-Id: I77ad7cded2a25da0eeb996f0a465be28120a887e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/314531
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
To reduce noise from proxy errors, downgrading
them to a warning and excluding them from error
reporting.
Change-Id: I1e1245ea534a6dd172587f9c13689b79fcf27001
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/306129
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
To avoid downloading a zip twice when processing a module,
implement a one-element zip cache in the proxy client.
The test in internal/worker/fetch_test.go explains in
more detail when this is useful.
For golang/go#44710
Change-Id: I89e67a5452833a7300fc2e10981e2b59594f10a9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/302532
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Information about when a symbol is added to the package is now presented
on the main unit page behind a feature flag.
For golang/go#37102
Change-Id: I21fc3aa7c6569b07d5b42409f6cb7db0e22dbf96
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/300673
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Symbol history is added to the versions page, for when a symbol is first
introduced to the package API.
For golang/go#37102
Change-Id: I22b7bc959a464dc38fe0bd39244c997fd51370f1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298309
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The proxy may have no version information even for a module that it
knows about.
For example:
```
> curl $GOPROXY/cloud.google.com/go/compute/metadata/@v/v0.0.0-20181107005212-dafb9c8d8707.info
{"Version":"v0.0.0-20181107005212-dafb9c8d8707","Time":"2018-11-07T00:52:12Z"}
> curl $GOPROXY/cloud.google.com/go/compute/metadata/@v/list
> curl -i $GOPROXY/cloud.google.com/go/compute/metadata/@latest
HTTP/2 410
```
If that happens, fetch.LatestModuleVersions returns nil and nothing is
added to the DB.
For golang/go#44710
Change-Id: I6710bbb643192f71c73424255153bc339be07a87
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298629
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Test is timing out in CI (can't repro locally).
Also add some logging.
Change-Id: I72ef35b94ff81c83fd2cd5ca5a7f69c9f31ce302
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295893
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Convert the proxy client to one which doesn't fetch at the beginning
of the request, so we don't have to pass a boolean argument through
several layers.
For golang/go#44437
Change-Id: I0495b5673cc399a1cdc16349d53ba4a9a4ecc380
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295889
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
When fetching a module at a version, the proxy datasource uses
fetch.RawLatestInfo to get the go.mod file at the raw latest version
of the module, then uses internal.RawLatestInfo.PopulateModule to
determine whether the module version is deprecated or retracted.
Also, add some proxy test modules to facilitate testing.
For golang/go#41321
For golang/go#43265
For golang/go#44437
Change-Id: I312346d72f656e598ad170135046ef85da8e9b11
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/295430
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
An error is fixed when creating the zipfile for std@master, which
prevented the module from being fetched. A test is also added.
For golang/go#44390
Change-Id: I279146aef10311822f47f16259a2223c879140c2
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/294831
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Remove the "Get" and "List" prefixes from proxy client method names.
The word "Get" is redundant for the proxy, since all you can do is get
stuff. It's also more idiomatic to avoid it when feasible.
Change-Id: I7d45fe06ab7a44083ff1ba75b284ec96fb785fe0
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/294631
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Instead of a special one-off method that sets the Disable-Module-Fetch
header, make it a property of the client.
This makes it easy to send the header for every request.
Change-Id: I4b5920d6852127e500e5eeaf187b3782db629bfb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/294630
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
If the proxy times out, don't call the Error-Reporting API.
Change-Id: Ia5e3f776e386b54f35d71e5839d0a6600708ca8d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/291451
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Add some examples that will be used in tests,
and provide a way to load them into proxy.Modules.
For golang/go#44214
Change-Id: Ia05b8d8e27071d821532e8747e1ab3b6e05085a0
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/290894
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Add some methods to the Module struct to make it easy to modify a
Module to create a variant for testing.
Move all Module-related code to a separate file.
For golang/go#44214
Change-Id: I4fb6026a7550641a0eda9d768e152cd65ce80b99
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/290893
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
The data we get from the errorreporting API isn't too useful
because the call to Report happens high on the stack. Try
to call Report closer to where the error happens.
Change-Id: I101fb4de0892c5d9967f6eb46d7c9cbd72fb567e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278952
Trust: Jonathan Amsterdam <jba@google.com>
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Add Client.GetInfoNoFetch, which sets the Disable-Module-Fetch
header to avoid fetching a module from the proxy if it isn't in
the proxy's cache. This reduces load on the proxy.
We return the new error code NotFetched instead of NotFound in this
case, so we can distinguish modules where this happened in the DB.
Change-Id: I2134d09e09b115e2ed59ba1a479ef20c1ebe4a7e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/277812
Trust: Jonathan Amsterdam <jba@google.com>
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Support requests for path@main in addition to path@master.
For golang/go#41312
Change-Id: Ie7665fbb3906e366c784a9cc592ffe37ef347671
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/274244
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Various uses of the variable version is renamed to requestedVersion for
clarity.
Change-Id: I542503a183f0b584689fe7fef6366e8555c3f84a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/256177
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Provide an efficient way to get the size of a module zip file
without downloading it.
For golang/go#41452
Change-Id: I9a996efb83146109a31e73b302af1b49f197b43f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/255737
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Avoid the call to GetInfo by requiring the version arg is already
resolved.
This was already true for the single non-test call to GetZip.
Change-Id: Id4992cffb9dd8b0b227cdcb9a5e71cc2345619e0
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/255577
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Several packages were missing package documentation, which is now added.
Change-Id: I1a88ebebaee0fddeb5a7bb9e2573375ad887c311
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/253607
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
If a fetch request failed due to a proxy timed out error, log this as an
INFO level log instead of an ERROR.
Change-Id: I2677a23e2dc9595f9040b78bec8d4be11b023068
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/251917
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
If zip.NewReader can't open the zip file, treat this as a bad module
(490) rather than a server error (500). That way we won't pointlessly
retry fetching the module.
Fixes b/166101385
Change-Id: I1c9999b09ff605b20cd420b43f016c7c0218c63a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/250237
Reviewed-by: Julie Qiu <julie@golang.org>
TestProxyServer is renamed to NewClientForServer, so that the function
doesn't have a "Test" prefix when it is not a test.
SetupTestProxy is renamed to SetupTestClient, since it returns a
proxy.Client, and proxy.SetupTestClient reads better.
Change-Id: I6c3152638e29b41e61275174df5e08e7eef5b2c4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/247297
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When the proxy times out on a request, it returns a status 404 with the
response text "not found: fetch timed out".
Rather than storing these as 404s in module_version_states, they are now
stored as 504, so that we will automatically retry fetching them.
Change-Id: I952bc9df8af2a108fffb04aef1c5fada98ccf0f7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245903
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
A Server is added to internal/proxy, which can be used to create a local
proxy server. In addition to the existing functionality of the test proxy,
Server supports the following:
- AddModule adds an additional module to the dataset of the proxy server
- The result of the latest and list endpoints are updated dynamically if
AddModule is called
- AddRoute adds an additional route to the proxy server
TestProxyServer now accepts a *Server instead of a *http.ServeMux. This
will be used for tests for frontend fetch in later CLs.
Change-Id: Iccc46f4c47d4b90bc1fd9bb061f2798c5dbb5b9d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245119
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Rather than validating the URL in proxy.New, assume that the URL that is
passed in is valid. This allows users to connect to a proxy running
locally in direct proxy mode.
For golang/go#40371
Change-Id: Id51cb27148987e58d214cef1c805b26b5138a6de
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245639
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
TestModule is renamed to Module, since it is shorter and proxy.Module is
equally descriptive as an exported variable.
Change-Id: Iebce664fa6e4580a4ab6b62fb73ad4bff611d067
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245118
Reviewed-by: Jonathan Amsterdam <jba@google.com>
A fetch endpoint is added to the frontend, which will queue a module to
be enqueued by the worker, if it doesn't already exist in the database.
After enqueuing, the fetch handler will poll the version_map table at a
constant rate, until the path returns or the request times out.
If the request fails, a corresponding statusCode and responseText will
be returned to be displayed to the user.
Updates golang/go#36811
Updates golang/go#37002
Updates golang/go#37106
Change-Id: Ic2e20146dc626bf296db05bc2abbfb50d6fd7991
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/743103
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
golang.org/x/discovery is renamed to golang.org/x/pkgsite.
When the repository is open sourced, it will be hosted at
go.googlesource.com/pkgsite.
Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273
Reviewed-by: Jonathan Amsterdam <jba@google.com>
proxy.TestModule now uses license testdata that already lives in
internal/testing/testdata.
Any TestModule that previously had a BSD3
license now has a BSD0 license, since the BSD0 license was already
present in internal/testing/testdata, and the type of
license for those tests did not matter, as long as it was redistributable.
Change-Id: Ic8837b2fe54e51e52086c859b1937de3269f8d2b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/710138
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Rather than having modules shared between packages, testdata now lives
in the package for which it is being used.
As a result, proxy.defaultModules is deprecated, and
proxy.SetupTestProxy requires any modules to be served to be passed as
args.
Change-Id: Ib307ab5a413cb254e8ceb97c8f36009f3b370c8d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/710008
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Testdata that is only used by the fetch package, but lives in
internal/proxy, is moved to internal/fetch.
Change-Id: I1cbc85e2efe060945c15a857b2230d193b4e5ac1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/710004
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Julie Qiu
Jonathan Amsterdam
Jamal Carvalho