The function was getting a bit long, so I split off the part
that finds files matching a build context.
That also allows better testing.
Change-Id: I231d671cf8615a7e71f6a1459bb0a7c692499e96
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/568935
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
The previous rule
.Documentation-source
was being overridden by
.Documentation h3 a
so I changed it to
.Documentation h3 a.Documentation-source
to make it more specific.
Change-Id: Ibe2b8d8f18a4e2bfc65244f651f9d5c612958731
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/567958
Reviewed-by: Julie Qiu <julieqiu@google.com>
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
Requests for packages/modules in standard library are now routed through
handleStdLib and use Go tags instead of semver.
Fixes b/142058801
Updates b/140191811
Change-Id: I452123a38e7f2263ee290e51411e95541f9052af
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/567738
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The source links from type and function names were incorrect
because they failed to include the path from the module root
to the package.
Change-Id: I264084e2f6527dfa86fa2462c60fd064c0334395
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/567959
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
details.go is split into the following files:
* handlers.go: contains handlers for /pkg and /mod
* header.go: contains code for creating a module or pkg header
* tabs.go: contains logic for routing tabs
Tests are also moved to the corresponding *_test.go files.
Change-Id: Ia0268e4a0cd61667cb63a9a2bc205147e4f7f194
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/567698
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
VersionForTag will be used in internal/frontend in the next
set of CLs for the new URL design.
Updates b/140191811
Change-Id: I85c11ac0d6e8c5e8c1040f2dbdc6a385a3e4ee09
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/567617
Reviewed-by: Jonathan Amsterdam <jba@google.com>
internal/etl.ParseVersionType is moved to internal.ParseVersionType
because it is used in internal/frontend and internal/etl.
Updates b/136649901
Change-Id: Ic071c8747c42014353ca38873b8cdd9c82d37804
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/567616
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Logic for fetchDocumentationDetails is moved from details.go to doc.go.
Change-Id: Ibafb918d4e9d81190e659028bbed7e61f712fd4b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/567619
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
There are several references to /pkg routes in our bugs and
screenshots. Redirect these URLs to the new package URL scheme.
Change-Id: I15cd8c43e9a20fba105b759bd36bcf885349c3f0
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/567620
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
An initial version of the url design described at go/go-discovery-urls
is implemented. The discovery site now accepts requests for packages at:
/<module-path>[@<version/<suffix>]
Requests for stdlib packages are accepted at:
/<pkgPath>@<version>
Requests for the stdlib is accept at:
/std
Additional work is needed for using Go tags instead of semantic versions
for stdlib packages, to redirect request to <pkg-path>@<version> to
<module-path>@<version>.
Updates b/140191811
Updates b/136484298
Change-Id: I42ba52177f2760a65399e361cddf302560ffa772
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/565996
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
GetPackageInModule is added to the DataSource struct and as a method to
proxydatasource.
GetImports is modified to accept modulePath in addition to pkgPath and
version as an argument.
Updates b/140191811
Updates b/136484298
Change-Id: Ibee233535a5970a2e4a5568181a4317a57a920eb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/565240
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
The columns name, synopsis, and license_types on the search_documents
table are now populated in UpsertSearchDocument.
Change-Id: I00ef98c30955da70a6b12fa729ca5d5642c24427
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/564566
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
An error occurred when GOOS and GOARCH return null values, since these
are not NOT NULL columns. This is now fixed.
Change-Id: Id22ebab19560114fb145a75313c01f78000ef5f3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/566236
Reviewed-by: Jonathan Amsterdam <jba@google.com>
GetPackageInModuleVersion is added, which fetches a package from the
database matching a package path, module path, and version combination.
Updates b/140191811
Updates b/136484298
Change-Id: Id5a31e0a32a048be4584bf81300aa6de9e2bdea1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/565301
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The previous popularSearcher(cutoff) search methods relied on partial
indexes to pre-filter search results to popular documents, and then
used the boundaries of our scoring function to determine whether such
partial searches were complete. This was a lot faster for popular search
terms, but still left something to be desired:
+ The partial index thresholds (8 importers and 50 importers) required
quite a lot of analysis to calibrate.
+ Postgres was not able to use the search document GIN index in the
search, so these popular searches were essentially scanning all
documents in their popular index. This resulted in a lot of extra
scanning when search results could have been satisfied by an even
smaller search.
In order to address these problems, we move away from declarative
queries (and playing games with the query planner), by using the
popular_search stored function to scan packages in descending order of
popularity, and exit as soon as possible.
Along the way, a few other trivial search methods are added to
optionally compare against the (now default) FastSearch.
An off-by-one error is fixed in hyperloglog estimation: we were
operating on the number of leading zeros, rather than the index of first
non-zero bit.
Updates b/141182438
Change-Id: Ib4c6bf16a456e603082d9c9e5c6d430e88879dbb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/564976
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
A popular_search stored function is added, that uses a cursor to
sequentially scan search documents, starting from the most popular
documents and proceeding in descending order of imported_by_count. This
allows the function to exit as soon as the page of results is provably
correct.
Updates b/141182438
Change-Id: I9d485522374c0f4b517e036163edf7671ad07da4
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/563877
Reviewed-by: Julie Qiu <julieqiu@google.com>
Module proxies may use 404 and 410 status codes interchangeably
to mean that the proxy does not have the module.¹
Remove the derrors.Gone error category, because at this time there
are no situations in which we want to treat it differently than
Not Found. Should that change in the future, we can re-add it.
Make the mapping from HTTP response status codes to errors more
explicit in the proxy client, since its needs are more specific
than the general derrors.FromHTTPStatus function provides.
¹ https://golang.org/cmd/go/#hdr-Module_downloading_and_verification
Change-Id: I47d4a736f81cebc345d5d0b8125d766d371a0349
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/565300
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
A custom search configuration golang is created, which will be used
for indexing and searching on the discovery site.
This search configuration is different from the default english search
config in the following ways:
* The url_path token type is remove, so that
'github.com/foo/bar@v1.2.3' is indexed only as the full URL string, and
not also '/foo/bar@v1.2.3'
* The asciiword token type is set to a 'simple,english_stem' mapping, so that
we can mix searched for "plural" words like "gopls" for an exact form of one
word and stemmed form for others. This will require updating to_tsvector calls to
use the golang search config, and to_tsquery to use a mix of 'simple' and 'english'.
Updates b/141930870
Change-Id: Id89fbea4d1024560a6dec5e7c058a1a5e192f8b8
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561880
Reviewed-by: Robert Findley <rfindley@google.com>
This change ensures there is no situation where the loop never exits.
Previously, this happened when the following call with invalid
parameters was made:
breadcrumbPath("golang.org/x/tools/notexist", "", "v0.0.0-20191002234911-9ade4c73f2af")
modPath is not expected to be the empty string. It happened due to
another problem elsewhere in the code (which was fixed in CL 564572).
Also improve the verb tense in a log message.
Change-Id: Id93f68da90c5285333b9f642de124bfe090e181c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/564571
Reviewed-by: Jonathan Amsterdam <jba@google.com>
- At fetch time, populate them with the values we use to parse the package.
- Save and load them from the DB.
Updates b/141852037.
Change-Id: If6972b4e2716be751b37c3fd5babbefadda005ec
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/563756
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
If a path given to /fetch is on the blacklist, return immediately and do not retry.
Updates b/142002842.
Change-Id: I71ffee0b98dbc16b73ce722741fe5e8a2877ffaa
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/564735
Reviewed-by: Robert Findley <rfindley@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
This table will hold a list of prefixes that we will not process or serve.
Updates b/142002842.
Change-Id: I3f4d67abff4fdf7b918055f7e94c442e0c48b138
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/564217
Reviewed-by: Robert Findley <rfindley@google.com>
We know the module path from an earlier call to findModule.
Include this information in the returned directory.
This helps render the breadcrumb for directories that
don't exist when using the direct proxy mode.
Change-Id: Ifbf26007e59919a1dc6942043734d543547371b6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/564572
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
A new btree index is added on search_documents, by descending
imported_by_count. This will be used in a new search method: a stored
procedure that scans packages in descending order of popularity.
Updates b/141182438
Change-Id: I6f4168e2b6e22e25fa2289bcdc0e2c23d0625595
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/563876
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
It is helpful to be able to at least temporarily test searches in the
frontend using different search methods. This CL adds a hidden query
param that allows executing the search using different query types.
Additionally, the pagination summary is updated to handle the case where
results counts are approximate.
Updates b/141182438
Change-Id: Ie56b2a7a5d47639025afb354042d048be52368cd
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/560267
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Now that hll columns are set when inserting search documents, we can
bulk populate them for existing documents.
Updates b/141182438
Change-Id: I12b8c1449c3c790329e3ddf88e030a9db1346602
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/560266
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The hyperloglog is implemented using the hll_register and
hll_leading_zeros columns on search documents, and integrated with fast
search results.
Updates b/141182438
Change-Id: I91e2b8c4599dea373df5c99cb4c10982a8616b57
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561156
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
The postgres function 'hashtext' is internal, and therefore really
shouldn't be relied upon. Also, it is only a 32 bit hash.
To work around this, we add a UDF implementing a 64 bit numeric hash
(taking the first 64 bits of the built-in md5 hash), and another that
can be used to count the number of leading zeros in a bigint.
Updates b/141182438
Change-Id: I33051273a41843370fec72b76bed37482617a5b3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561155
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Currently, we need to execute a second query to get various
package metadata when generating search results. This metadata includes:
* synopsis
* license_types
* name
These columns are now added to the search_documents table, so that we
don't have to execute an additional query.
Change-Id: Ia4e30c0688ba81a8742c03a7b0c2872ea5aa9930
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/564564
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
If a module has a package that has .go files, but we can't process the
package because of limitations of our system, mark that fact in the
module_version_states table with a special status.
We choose a 2xx status because the module processing succeeded, but
not 200 so we can distinguish these modules later for reprocessing
if our limitations change (e.g. if we add a build environment).
We choose a non-existent HTTP status (290) to avoid confusion
with any existing status code.
Updates b/133187024.
Change-Id: I1ec1ce7a6d59287c41fcd4d7dacf9ca4b8fd5f2c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/564216
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Specify that we are using the language/compiler for Go 1.12, because that's
our production version.
Change-Id: I48e56c16227eb9869806699d5fb7d16ae876f0f7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/564222
Reviewed-by: Julie Qiu <julieqiu@google.com>
It's possible that a vanity import path's meta tags mention
a known repo pattern only in the URL templates. This is what gopkg.in
tags do. For example:
<meta name="go-import" content="gopkg.in/yaml.v2 git https://gopkg.in/yaml.v2">
<meta name="go-source" content="gopkg.in/yaml.v2 _
https://github.com/go-yaml/yaml/tree/v2.2.3{/dir}https://github.com/go-yaml/yaml/blob/v2.2.3{/dir}/{file}#L{line}">
In this case, we can match the templates in the tag to our list of
known patterns, and use the version-aware templates we get from that
match instead of the ones in the tag. Here, for example, we can note
that the source lives on GitHub and use the pattern that has
"{commit}" instead of the hardcoded "v2.2.3", making it possible to
refer to other versions of the module.
Change-Id: Ia998f47ec52996eb7f71377624c638bedfe1994f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562704
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Remove "vN" when constructing the module directory.
- Distinguish subdirectory and branch cases by looking for a go.mod file.
Fixes b/141769404.
Change-Id: I20328ace5257c2102c0248e6f92bdce2debe94e5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562701
Reviewed-by: Julie Qiu <julieqiu@google.com>
gitlab.com serves source information just like github. Add it to the list
of known patterns.
Change-Id: I0b102d2c56c03a4ec8904f089f09ef6793db9c7f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562177
Reviewed-by: Julie Qiu <julieqiu@google.com>
Use the go command's meta-tag mechanism to find source code
for vanity module paths.
If a module path doesn't match one of our built-in patterns,
make a GET request to the path and parse the go-import and
go-source meta tags on the page to learn where the source
lives and how to link to it.
This process is slightly different than godoc.org's (as implemented
in go.googlesource.com/gddo):
- godoc.org uses package import paths; we use module paths instead.
That can result in different behavior when a module doesn't
correspond to a package and no tags are served at the module path
URL, even though its package URLs do serve tags. The discovery site
will not show source links for those packages. This is a genuine
breaking behavior change, but we're going to make it because serving
tags at the module level is the right thing to do, and we can easily
fix the handful of modules where this is a problem.
- godoc.org needs to fetch the code, so it needs information from the
tags like the vcs, and it needs to keep the go-import and go-source
tags separate (the first being for fetching, the second for
linking). Since the discovery site already has the code from the zip
file, we just need go-source information, using go-imports as a
fallback.
- The go-source tag contents don't provide a way to insert a version,
so they're not usable as-is for linking to versioned code. For now,
we ignore the URL templates in the tag and just try to match the
repo. So a vanity URL that redirects to GitHub will work fine, since
we know how to link to files in GitHub; but one that redirects to
some random code-hosting site will not have source links on the
discovery site, even though it might on godoc.org.
Fixes b/141771975.
Change-Id: Ie07bff3a77350cdaa7e45af0c7e5bce0a922e945
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562176
Reviewed-by: Julie Qiu <julieqiu@google.com>
After etl parses files, use use source.Info to generate links from the
ast.Nodes.
Add that function to the template that dochtml uses, and call it
on types and function names.
Updates b/140431331.
Change-Id: I2494a1b20e137b11d4cc35b52e53effc93fcded1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561218
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add internal/source, which will know how to generate URLs that
refer to the source code or other files inside modules.
This package will be used to link to module and package source code,
and also for relative links in README files.
For better testing, introduce the httpreplay package. With -record, it
makes HTTP requests to the actual servers and records the responses.
With no flags, it replays the recordings.
Fixes b/141771930.
Updates b/140431331.
Change-Id: I72dcd5535b8be7c6b27dd89245a048d23fd14160
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561217
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
A bug where "example.com/abcd" was incorrectly considered a
subdirectory of "example.com/ab" is now fixed.
Fixes b/141128705
Change-Id: I18a8a76b39c6ababc157348b5455bdb85c94e94e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561350
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Currently, the package details URL will contain the original search
query if it was arrived at from the search page. This is now removed, since
users want to be able to copy and paste the URL without first having to
make it shorter.
Fixes b/140329357
Change-Id: I8ae78d82667c8f721e487799af1f25e69fae45cf
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561352
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Additionally, class="Header" when used in details.tmpl is renamed to
class="DetailsHeader", to avoid confusion when class="Header" is used in
base.tmpl.
Fixes b/141545555
Change-Id: I1680335a3458ac5be0ebe40ec7add580c1157067
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561735
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Currently, the error page for status SeeOther will say "MODULE <path>" or
"PACKAGE <path>". It now says "Module <path>" or "Package <path>".
Change-Id: I5e75229d7d69ea79a83a38d9d0d53ff6bd70afb1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561172
Reviewed-by: Robert Findley <rfindley@google.com>
Jonathan Amsterdam
Julie Qiu
Rob Findley
Dmitri Shuralyov