The source for these import paths is served by the git-arr web
viewer, which doesn't yet handle tags. Until it does, serve
the source from master.
For golang/go#44607
Change-Id: I94047a887ea6f1038ae812864ad343876b74ff8e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303609
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Remove the ".git" from the repo URL of source templates
when we know it will result in invalid URLs.
The ".git" is understood by the Go command, but not
by some code hosting sites. To be safe, only remove
it on code hosting sites we've tested. For now,
that's GitHub and GitLab.
Remove the suffix in three places:
1. When we construct a SourceInfo from the repo URL and meta
tags. This will fix newly processed modules.
2. When we unmarshal a SourceInfo from the DB. This will fix it during
rendering for already-processed modules. It will save us from having
to reprocess or backfill.
3. In the function we use in tests.
Fixesgolang/go#44032
Change-Id: Ic6d48d0422dc12ce8dbc4586f8d6c586f463531e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/288612
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Previously, when we encountered a repo whose URL doesn't match an
existing pattern, we did not generate any URL templates for it,
meaning we could not render source links in the documentation.
This CL uses the templates in the go-source meta tag to guess the
version-aware templates that are likely to work for the repo.
For golang/go#40477
Change-Id: I2d1978da5a6de1af19284233dbab9ac1ae2cb582
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/285312
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Add NewClientForTesting, which returns a Client that does not
make any HTTP requests.
The practical effect is to eliminate a lot of errors from test
logs in the internal/fetch package.
Change-Id: Ie47ceb1a8984d4408ea361baefa1610aeed1b02a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/279793
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Link to source files in the canonical Go repo, go.googlesource.com/go.
Continue to serve raw content (for images) from the GitHub mirror,
since we can't from googlesource.com.
Fixesgolang/go#43180
Change-Id: Iac513d6481bcdb73ae73eed40d33518ffd3a0158
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278293
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
source.ModuleInfo is incorrectly documented as source.LegacyModuleInfo,
which is now fixed.
A comment in RawURL is also fixed.
For golang/go#39629
Change-Id: I30b650f2895970e36446659a02a6f3e8e54ef578
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/258301
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Trust: Julie Qiu <julie@golang.org>
Add support for a set of packages that defer to another website to
display their source code. To be able to produce expected URLs, we
need to add two more URL template variables:
• {importPath} - Package import path ("example.com/myrepo/mypkg").
• {base} - Base name of file containing the identifier,
including file extension ("file.go").
Also add a new optional Repo URL template for overriding the home
page of the repository, which is something that was possible with
the original go-source meta tag.
Document the existing URL template variables, so that it's easier
to understand how to use them.
For golang/go#40477.
Change-Id: I70b857155f69c5c3ed41e78daccb90153a927290
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/246239
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Treat any repo URL beginning with "gitea." as if it uses the Gitea
system.
Also, sanity-check templates and fix some bugs.
Change-Id: Ib6c3ca028d57e05504faef4bfdf6aa4439525fb7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245379
Reviewed-by: Julie Qiu <julie@golang.org>
It's not important whether certain patterns are part of the go command
or not.
Change-Id: I8e921b0f2256efa419b3df41a9d1925cb0fa8cd7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245037
Reviewed-by: Julie Qiu <julie@golang.org>
Now that the entire server uses safehtml, it is no longer necessary
to do anything special with the user-provided values in this package.
Every use of the Info.XXXURL methods is part of rendering a safe
template, so all the escaping will happen then.
Change-Id: I5e64b08d4c7b8e7fd56cf1a0bed49691f2a8ee82
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242917
Reviewed-by: Julie Qiu <julie@golang.org>
Replace the previous github template for raw data
(raw.githubusercontent...) with a new one that is more uniform (the
same as the blob template, with "raw" instead of "blob").
The main advantage is that we can drop the "repoPath" parameter to the
template, which was necessary only to support the unusual GitHub
URLs.
Now that GitHub and GitLab templates are identical, we can drop the
GitLab ones. We preserve "gitlab" in the map from kinds to templates
so we can continue to unmarshal DB modules.source_info columns
that use "gitlab".
Change-Id: Id3d45929f0eb2c040109449d4208279cca44f079
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243217
Reviewed-by: Julie Qiu <julie@golang.org>
There has been progress made since the package comments
were written. Update them to reflect the current state.
For golang/go#39621.
Change-Id: I3f950428a96bc0561b45b0d49117d2eda58d3cf7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240187
Reviewed-by: Julie Qiu <julie@golang.org>
Escape any strings that may come from outside the program and will
be rendered as HTML.
Also, fix some typos.
Also, put args to a cmp.Diff call in the right order.
Change-Id: I1fc6013e200326af8719b14c28a607fc7936e20a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239478
Reviewed-by: Julie Qiu <julie@golang.org>
This is the first CL to clean up various TODOs that are outdated, and
replace internal issue links with GitHub issue links.
Updates golang/go#39621
Change-Id: If270e8b2e8198c007cb4aa71ad8486182f4f3380
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238319
Reviewed-by: Jonathan Amsterdam <jba@google.com>
We were creating a reply client, but not passing it in, so
TestModuleInfo was actually hitting the internet.
Fixes b/158850091.
Change-Id: I4e981299a538750cb519c441e05805b29c7925a8
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/769682
Reviewed-by: Julie Qiu <julieqiu@google.com>
ModuleInfo is renamed to LegacyModuleInfo, as a step towards
deprecating LegacyReadmeFilePath and LegacyReadmeContents.
In a follow up CL, we will add ModuleInfo as an embedded struct
to LegacyModuleInfo.
Change-Id: Ie452420448eec1d13edaf62e548df0b9e2cbbe4b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766479
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The following structs have been renamed with a Legacy prefix:
* internal.Directory
* internal.Package
* internal.VersionedPackage
* sample.Package
The following fields on internal.ModuleInfo have also been changed:
* ReadmeFilePath
* ReadmeFileContents
This is done to help us distinguish between legacy and method
structs/methods while migrating code to the new data model.
Change-Id: Ibedf71d4db6323ef5aa05d73a0240537ea6073d3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/765160
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Running with `go test -record` revealed some 404s (pages that used to
exist but no longer do). I replaced them with similar pages.
Change-Id: I6903229e8221923e22338bdae3f112308aecd19e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/756267
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Additional traces are added to the fetch process so that we can get more
detailed information on what is taking up time in a fetch process.
Change-Id: I8f74a0d2117a18002d64453d222cb08a1fb81ff2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/729482
Reviewed-by: Jonathan Amsterdam <jba@google.com>
golang.org/x/discovery is renamed to golang.org/x/pkgsite.
When the repository is open sourced, it will be hosted at
go.googlesource.com/pkgsite.
Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273
Reviewed-by: Jonathan Amsterdam <jba@google.com>
sourceClient wasn't being initialized by worker.NewServer, which led to
a nil pointer error that is now fixed.
Change-Id: I5d16e69e60c9a1ba489a0b5ec0e3a1a6a1f19366
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/711165
Reviewed-by: Jonathan Amsterdam <jba@google.com>
proxy.TestVersion is deprecated in favor of proxy.TestModule.
TestModule represents a module version used to generate testdata.
Change-Id: I718045917a352af218ccebb45fa0d2875a5b9610
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/709704
Reviewed-by: Jonathan Amsterdam <jba@google.com>
source.Client is added, which replaces the used of http.DefaultClient
when fetching source code. This also allows us to set a custom timeout
for tests vs when running the worker.
Change-Id: I5b7b0fd32fa7a2cf836b951af1ad471751fade00
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/709198
Reviewed-by: Jonathan Amsterdam <jba@google.com>
- The requestlog middleware gets the trace ID from the request and
adds it to the request's context.
- The internal/log package retrieves the traceID and adds it to log
messages. It also creates two logs instead of one, a "parent" log
used in requestlog, and a "child" log used for all other logging.
Together, these two changes will cause the Stackdriver log viewer to
group all log messages for a request with the request start and end
log messages.
- These changes require that all log functions take a context, so I
plumbed one through everywhere. In a handful of cases it didn't seem
worth doing the plumbing. I used context.TODO() for those so we can
easily find and re-evalaute them.
Change-Id: I663588463520187d0549a8f802ba9cb44a893592
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/622940
Reviewed-by: Robert Findley <rfindley@google.com>
Now that we are on Go 1.13, we can use the errors package and remove
xerrors.
Change-Id: I8b30bcea97da25cc92b8e250299a68417a85143d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/623768
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Some sites (gonum.org in particular) return an error status (like 404)
to a ?go=get=1 URL, but also serve a body with meta tags. Ignore the
404 for the http request.
Fixes b/144596157.
Change-Id: Ib59177b6847890975768de8fe4b623e188828f1f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/622164
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Store the output of licensecheck.Cover in the licenses table.
This will make our lives easier if we ever want to revisit our
coverage parameters.
In the process, generalize scanning of jsonb columns
into Go values.
Change-Id: Iadc89c89c2590e0894fefcdcd461283d90b2e701
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/601900
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
The module directory we use to create links to the standard module's
source is "src". That is only partly true, however: the .go files are
there, but the README is at the repo root. This matters when
constructing raw links from the README.
We add a special check for this case in Info.RawURL.
Note that this is a bit fragile and hacky: if we ever wanted a raw
link to a source file, for example, the URL would be wrong. But the
only alternative I can see now is to make everything more complicated
just for this one case, which isn't worth it.
Fixes b/143208537.
Change-Id: Ide47b3739eaaf9f6b7e9b4631811ca3695353bb3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/584337
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
If a Readme has a relative link that is not an image, convert
it to an absolute link to the file using source info.
Fixes b/140250096.
Change-Id: I9ca629963ca87d7605c0217e49671b33c7dd0f86
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/576296
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Change translateRelativeLink to use the source.Info.RawURL method.
- Remove internal.VersionInfo.RepositoryURL; it is redundant.
- Remove unnecessary tests from overview_test.go; they no longer make
sense or are effectively tested in internal/source.
Change-Id: Ibcf7f120ca1d257178b84ee69d22ca256de0b766
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/575158
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Add a SourceInfo field to VersionInfo.
- Add functions to convert a source.Info to and from JSON.
- Modify postgres statements to read and write from the DB.
Keep the VersionInfo.RepositoryURL field around for now to avoid
messing with the frontend.
Updates b/132219298, b/138752200.
Change-Id: I6ebb3af3afa14fda92c2804ab2d0294742a3bf41
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/575156
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Move some version-related code like IsPseudoVersion from internal to a
new package.
This is needed to break an import cycle:
- internal.VersionInfo needs a *source.Info field
- internal/source needs to call internal.IsPseudoVersion
Change-Id: I125bd019bc2ec9cec9d1ec0945bbaafdfecc5cfc
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/573775
Reviewed-by: Julie Qiu <julieqiu@google.com>
- source.ModuleInfo gains an http.Client arg
- ETL tests set it to something that immediately returns an error
Change-Id: I6cab563bc22424ab7037c6827309285259b7dd9c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/574055
Reviewed-by: Robert Findley <rfindley@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Add support for a URL template that can produce URLs for raw
file content. To be used to generate relative links in READMEs.
Change-Id: I6f6e1662825b96ddccd9dbcc4e48fbc92e7b0d82
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/572879
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
git.apache.org repos are hosted on GitHub. The final ".git" in the module
path throws off the meta-tag matcher, so handle it specially.
Change-Id: I57a8da4d4a000f0acd72945b6efa4439579e6518
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/572877
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Change some log messages to Info level, because they aren't
actionable server errors.
Remove some messages because they're redundant.
- Unless the error code from a fetch is 5xx, log at the info level.
- Don't treat missing source templates as errors. (Based on going through
the logs, we've identified all major patterns for source links.)
- Remove some log messages when successfully processing a package: they
are redundant.
This avoids cluttering the Error-level logs with events that aren't
our problem, like a module having no packages (for some reason there
are many "modules" in the index that contain no Go code).
Change-Id: Iff2f78cc87150a3d705837d069149f7cfdbfefcb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/570674
Reviewed-by: Robert Findley <rfindley@google.com>
Add two more patterns to our hard-coded list:
- gitee.com is a GitHub look-alike, and links behave the same.
- We'll assume any host of the form "gitlab.*" (e.g. "gitlab.ghn.vn") behaves
like gitlab.com, which itself behaves like git.
Change-Id: I4e4110699086ecd303ec4a4e5a882fb66f36c3b9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/570673
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
The source links from type and function names were incorrect
because they failed to include the path from the module root
to the package.
Change-Id: I264084e2f6527dfa86fa2462c60fd064c0334395
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/567959
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
It's possible that a vanity import path's meta tags mention
a known repo pattern only in the URL templates. This is what gopkg.in
tags do. For example:
<meta name="go-import" content="gopkg.in/yaml.v2 git https://gopkg.in/yaml.v2">
<meta name="go-source" content="gopkg.in/yaml.v2 _
https://github.com/go-yaml/yaml/tree/v2.2.3{/dir}https://github.com/go-yaml/yaml/blob/v2.2.3{/dir}/{file}#L{line}">
In this case, we can match the templates in the tag to our list of
known patterns, and use the version-aware templates we get from that
match instead of the ones in the tag. Here, for example, we can note
that the source lives on GitHub and use the pattern that has
"{commit}" instead of the hardcoded "v2.2.3", making it possible to
refer to other versions of the module.
Change-Id: Ia998f47ec52996eb7f71377624c638bedfe1994f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562704
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Remove "vN" when constructing the module directory.
- Distinguish subdirectory and branch cases by looking for a go.mod file.
Fixes b/141769404.
Change-Id: I20328ace5257c2102c0248e6f92bdce2debe94e5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562701
Reviewed-by: Julie Qiu <julieqiu@google.com>
Jonathan Amsterdam
Julie Qiu
Dmitri Shuralyov