Treat any repo URL beginning with "gitea." as if it uses the Gitea
system.
Also, sanity-check templates and fix some bugs.
Change-Id: Ib6c3ca028d57e05504faef4bfdf6aa4439525fb7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245379
Reviewed-by: Julie Qiu <julie@golang.org>
Now that the entire server uses safehtml, it is no longer necessary
to do anything special with the user-provided values in this package.
Every use of the Info.XXXURL methods is part of rendering a safe
template, so all the escaping will happen then.
Change-Id: I5e64b08d4c7b8e7fd56cf1a0bed49691f2a8ee82
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242917
Reviewed-by: Julie Qiu <julie@golang.org>
Replace the previous github template for raw data
(raw.githubusercontent...) with a new one that is more uniform (the
same as the blob template, with "raw" instead of "blob").
The main advantage is that we can drop the "repoPath" parameter to the
template, which was necessary only to support the unusual GitHub
URLs.
Now that GitHub and GitLab templates are identical, we can drop the
GitLab ones. We preserve "gitlab" in the map from kinds to templates
so we can continue to unmarshal DB modules.source_info columns
that use "gitlab".
Change-Id: Id3d45929f0eb2c040109449d4208279cca44f079
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243217
Reviewed-by: Julie Qiu <julie@golang.org>
Escape any strings that may come from outside the program and will
be rendered as HTML.
Also, fix some typos.
Also, put args to a cmp.Diff call in the right order.
Change-Id: I1fc6013e200326af8719b14c28a607fc7936e20a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239478
Reviewed-by: Julie Qiu <julie@golang.org>
We were creating a reply client, but not passing it in, so
TestModuleInfo was actually hitting the internet.
Fixes b/158850091.
Change-Id: I4e981299a538750cb519c441e05805b29c7925a8
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/769682
Reviewed-by: Julie Qiu <julieqiu@google.com>
Running with `go test -record` revealed some 404s (pages that used to
exist but no longer do). I replaced them with similar pages.
Change-Id: I6903229e8221923e22338bdae3f112308aecd19e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/756267
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
proxy.TestVersion is deprecated in favor of proxy.TestModule.
TestModule represents a module version used to generate testdata.
Change-Id: I718045917a352af218ccebb45fa0d2875a5b9610
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/709704
Reviewed-by: Jonathan Amsterdam <jba@google.com>
source.Client is added, which replaces the used of http.DefaultClient
when fetching source code. This also allows us to set a custom timeout
for tests vs when running the worker.
Change-Id: I5b7b0fd32fa7a2cf836b951af1ad471751fade00
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/709198
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Some sites (gonum.org in particular) return an error status (like 404)
to a ?go=get=1 URL, but also serve a body with meta tags. Ignore the
404 for the http request.
Fixes b/144596157.
Change-Id: Ib59177b6847890975768de8fe4b623e188828f1f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/622164
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
The module directory we use to create links to the standard module's
source is "src". That is only partly true, however: the .go files are
there, but the README is at the repo root. This matters when
constructing raw links from the README.
We add a special check for this case in Info.RawURL.
Note that this is a bit fragile and hacky: if we ever wanted a raw
link to a source file, for example, the URL would be wrong. But the
only alternative I can see now is to make everything more complicated
just for this one case, which isn't worth it.
Fixes b/143208537.
Change-Id: Ide47b3739eaaf9f6b7e9b4631811ca3695353bb3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/584337
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Add a SourceInfo field to VersionInfo.
- Add functions to convert a source.Info to and from JSON.
- Modify postgres statements to read and write from the DB.
Keep the VersionInfo.RepositoryURL field around for now to avoid
messing with the frontend.
Updates b/132219298, b/138752200.
Change-Id: I6ebb3af3afa14fda92c2804ab2d0294742a3bf41
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/575156
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
- source.ModuleInfo gains an http.Client arg
- ETL tests set it to something that immediately returns an error
Change-Id: I6cab563bc22424ab7037c6827309285259b7dd9c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/574055
Reviewed-by: Robert Findley <rfindley@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Add support for a URL template that can produce URLs for raw
file content. To be used to generate relative links in READMEs.
Change-Id: I6f6e1662825b96ddccd9dbcc4e48fbc92e7b0d82
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/572879
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
git.apache.org repos are hosted on GitHub. The final ".git" in the module
path throws off the meta-tag matcher, so handle it specially.
Change-Id: I57a8da4d4a000f0acd72945b6efa4439579e6518
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/572877
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add two more patterns to our hard-coded list:
- gitee.com is a GitHub look-alike, and links behave the same.
- We'll assume any host of the form "gitlab.*" (e.g. "gitlab.ghn.vn") behaves
like gitlab.com, which itself behaves like git.
Change-Id: I4e4110699086ecd303ec4a4e5a882fb66f36c3b9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/570673
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
It's possible that a vanity import path's meta tags mention
a known repo pattern only in the URL templates. This is what gopkg.in
tags do. For example:
<meta name="go-import" content="gopkg.in/yaml.v2 git https://gopkg.in/yaml.v2">
<meta name="go-source" content="gopkg.in/yaml.v2 _
https://github.com/go-yaml/yaml/tree/v2.2.3{/dir}https://github.com/go-yaml/yaml/blob/v2.2.3{/dir}/{file}#L{line}">
In this case, we can match the templates in the tag to our list of
known patterns, and use the version-aware templates we get from that
match instead of the ones in the tag. Here, for example, we can note
that the source lives on GitHub and use the pattern that has
"{commit}" instead of the hardcoded "v2.2.3", making it possible to
refer to other versions of the module.
Change-Id: Ia998f47ec52996eb7f71377624c638bedfe1994f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562704
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Remove "vN" when constructing the module directory.
- Distinguish subdirectory and branch cases by looking for a go.mod file.
Fixes b/141769404.
Change-Id: I20328ace5257c2102c0248e6f92bdce2debe94e5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562701
Reviewed-by: Julie Qiu <julieqiu@google.com>
gitlab.com serves source information just like github. Add it to the list
of known patterns.
Change-Id: I0b102d2c56c03a4ec8904f089f09ef6793db9c7f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562177
Reviewed-by: Julie Qiu <julieqiu@google.com>
Use the go command's meta-tag mechanism to find source code
for vanity module paths.
If a module path doesn't match one of our built-in patterns,
make a GET request to the path and parse the go-import and
go-source meta tags on the page to learn where the source
lives and how to link to it.
This process is slightly different than godoc.org's (as implemented
in go.googlesource.com/gddo):
- godoc.org uses package import paths; we use module paths instead.
That can result in different behavior when a module doesn't
correspond to a package and no tags are served at the module path
URL, even though its package URLs do serve tags. The discovery site
will not show source links for those packages. This is a genuine
breaking behavior change, but we're going to make it because serving
tags at the module level is the right thing to do, and we can easily
fix the handful of modules where this is a problem.
- godoc.org needs to fetch the code, so it needs information from the
tags like the vcs, and it needs to keep the go-import and go-source
tags separate (the first being for fetching, the second for
linking). Since the discovery site already has the code from the zip
file, we just need go-source information, using go-imports as a
fallback.
- The go-source tag contents don't provide a way to insert a version,
so they're not usable as-is for linking to versioned code. For now,
we ignore the URL templates in the tag and just try to match the
repo. So a vanity URL that redirects to GitHub will work fine, since
we know how to link to files in GitHub; but one that redirects to
some random code-hosting site will not have source links on the
discovery site, even though it might on godoc.org.
Fixes b/141771975.
Change-Id: Ie07bff3a77350cdaa7e45af0c7e5bce0a922e945
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562176
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add internal/source, which will know how to generate URLs that
refer to the source code or other files inside modules.
This package will be used to link to module and package source code,
and also for relative links in README files.
For better testing, introduce the httpreplay package. With -record, it
makes HTTP requests to the actual servers and records the responses.
With no flags, it replays the recordings.
Fixes b/141771930.
Updates b/140431331.
Change-Id: I72dcd5535b8be7c6b27dd89245a048d23fd14160
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561217
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Jonathan Amsterdam
Julie Qiu