This unifies license-related functionality into a new internal/license
package, with the intention of making it easier to understand handling
of licenses. It also makes some type names a little cleaner.
A few minor changes are made along the way (e.g. LicenseInfo ->
Metadata), and tests are added for license matching.
Updates b/131921712
Updates b/131741680
Change-Id: I6a766513e971cea74938fbc60f0220c702e66bcd
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/459926
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add new IndexVersion and VersionState types, along with functions for
storing, updating, and retrieving them from postgres.
Updates b/131614050
Change-Id: Ifbc26b061bbdc86c8c727d948f7b4f075d405cb2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/459933
Reviewed-by: Julie Qiu <julieqiu@google.com>
Search results are currently indicated as "<start> of <total>" (such as
"0 of 95").
They are now displayed as <start>-<end> of <total> (such as "1-10 of
95").
Fixes b/131929222
Updates b/131908602
Change-Id: I266d7094a261b8f3674b6e2368c618c4d6a73221
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/460700
Reviewed-by: Robert Findley <rfindley@google.com>
The pages on the module view are now listed in order
of the package path to preserve directory structure.
Fixes b/131859852
Change-Id: I1924fd48c310e981e5659a9a3e5c44bb1e6bde1f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/459561
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
The mvw_search_documents materialized view is now used for search.
postgres.RefreshSearchDocuments is introduced, which will be used
to refresh data in the materialized view in a later CL.
Fixes b/131861129
Fixes b/131859794
Fixes b/131911629
Updates b/131908602
Change-Id: I33209b65b7d00420660bd6dc58b03250c9fc144a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/459559
Reviewed-by: Robert Findley <rfindley@google.com>
The search query has been optimized with the following:
* Using a where clause to filter out tsv_search_tokens
* Inline query to get packages and licenses and join with the versions
table in a subquery
This reduced a search for 'cloud' from 7934.626 ms to 1746.115 ms.
Updates b/130090305
Change-Id: I6c71800db9055e02474bfe30d4c9d2ee1c4f7e8a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/457637
Reviewed-by: Robert Findley <rfindley@google.com>
At the moment, there is no limit on the number of page links on the
search page, which becomes messy when search has a lot of results.
Page numbers are now calculated using pagesToLink, which returns an
integer slice representing page numbers that will displayed. It
optimizes for the current page to be in the middle of that range
(similar to google.com).
Fixes b/131836875
Updates b/131862035
Change-Id: I9dafc35bff2278a4850bbded086a0000b5f76d3d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/458088
Reviewed-by: Robert Findley <rfindley@google.com>
A bug was introduced in commit cc073d7624cfc3ca6fd8641101a5ffcc6bd1996d, causing
there to be two schema migrations with 000042.
000042_add_module_version_state is renumbered to 000043_add_module_version_state.
Change-Id: Ib25c631d5c8f662c6f8e8aa5c4da6fc395f7298e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/459560
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Add a module_version_state table to implement the ETL state for new
module versions.
Updates b/131614050
Change-Id: I7496b428588b1d4f461c5da7376012e8513d842a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456916
Reviewed-by: Julie Qiu <julieqiu@google.com>
The materialized view mvw_search_documents is created to cache data
needed for generating search results. It contains information about the
number of importers for each package, tsvector tokens used to search,
and package metadata.
This table will be used by internal/postgres.Search in a later CL.
Updates b/131861129
Change-Id: Ib021c1e5cdec07f18304ff8ec5f7198419887dd8
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/458701
Reviewed-by: Robert Findley <rfindley@google.com>
The output is not actionable because thirdparty packages are copied
from an external source, and we can't modify them directly in this
project.
Remove trailing slash from a relative import path "./internal/secrets".
It's more common to not include a trailing slash when specifying import
paths.
Fixes b/131821500
Change-Id: Id7458b1474a4218422035cb15e305e2b2a356b64
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/457840
Reviewed-by: Robert Findley <rfindley@google.com>
At least temporarily, relax the content security policy to allow
arbitrary images, since this is currently breaking images in README
rendering.
Updates b/130541353
Change-Id: Ifbdea9c5600851f998553b413916ddab3d7ffd43
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/459719
Reviewed-by: Andrew Bonventre <andybons@google.com>
This change is the first step in applying proper structure
and styling based on mockups from UX.
The template structure has changed with a single "base" page,
base.html used as the first template parsed when constructing
each page. Other templates have been updated to use this new
structure.
An option has been added to the frontend controller to reload
templates on each request to ease development.
Change-Id: I6856ad3f854249aaf29e46fc6c3e563039ab931d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/457866
Reviewed-by: Julie Qiu <julieqiu@google.com>
Update to a newer versions of github.com/golang-migrate/migrate/v4,
cloud.google.com/go, github.com/google/go-cmp, gocloud.dev, and
google.golang.org/genproto modules in order to significantly reduce
the size of the transitive closure of the required modules (from 640
down to 362).
The latest released version of gocloud.dev module is currently v0.13.0,
and does not yet include the change to drop many unnecessary modules
(specifically, see the go.mod/go.sum file diffs in the commit
4a87797b25).
The next release version v0.14.0 of gocloud.dev will include that
improvement, and it's going to come out within a month. We can update
to it then. All tests pass with the new dependency versions.
This reduces the deploy time of ./cmd/frontend (without a module proxy)
from 9-10 minutes to 2-3 minutes. The deploy time can be improved even
further by using a proxy.
Also update to new versions of htmlg and component packages that have
more visible license information.
Change-Id: I12851ac7a3e66f1d66348d83239686375466ea18
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/457839
Reviewed-by: Robert Findley <rfindley@google.com>
Add a ContentSecurityPolicy middleware to the frontend. For now, this
adds just a basic policy; the associated bug will not be closed until we
have finalized the policy.
Updates b/130541353
Change-Id: I6da1724525eacc02ad334a15e09c226f28ba6001
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/457874
Reviewed-by: Andrew Bonventre <andybons@google.com>
Clearly I've missed a couple warnings from our presubmit, so I'm making
it louder:
+ Use terminal colors to call out errors and warnings.
+ Always run header check on all (known) internal files, so that it
doesn't go silent after a bad commit is merged.
Change-Id: I88116e1dcb0ed6c15b8c1369cbb37de00bd57efd
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/457641
Reviewed-by: Julie Qiu <julieqiu@google.com>
This CL introduces an initial package documentation rendering code
for the discovery website. It performs the rendering during package
processing in the fetch service. There are many minor details left
to tweak, but it has the right general shape.
Also reduce the scope of work done by packages.Load to work on the
extracted module, and not try to pull in its transitive closure of
dependencies from the internet. What's left is to come up with a way
of writing a test for this, so the behavior doesn't regress.
Load Work Sans, Roboto, and Source Code Pro fonts for use on package
documentation page.
Update x/tools module to latest version.
Change-Id: I08fbbea8ef2b964c7b083a6dae3a725d2b7e4b68
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/457636
Reviewed-by: Julie Qiu <julieqiu@google.com>
The following indexes are added:
* idx_semver_sort ON versions (module_path, major DESC, minor DESC, patch DESC, prerelease DESC)
* idx_package_licenses ON package_licenses(version, module_path, file_path)
* idx_imports_to_path ON imports(to_path, from_path)
The following columns are altered to TYPE TEXT COLLATE "C", since they are used for sorting
* documents.module_path
* documents.package_path
* documents.series_path
* licenses.file_path
* licenses.module_path
* licenses.module_path
* package_licenses.module_path
* package_licenses.package_path
The primary key for imports is ordered to (to_path, from_path, from_version) to improve
performance for aggregating importedby count.
Fixes b/129600165
Updates b/130090305
Change-Id: Ia158884e5eaab1e5b49a651af38a5439b961052e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/457638
Reviewed-by: Robert Findley <rfindley@google.com>
If switching between branches with different migrations, it's likely
that migration.Up will fail on your test databases. This can be a pain
now that there are multiple test databases to fix.
To handle this, automate the re-creation of the database if the
migration fails during the Up step.
Also, remove no-longer-needed migration for discovery-database-test.
Updates b/130719094
Change-Id: Ibb0550b95c2e7a3e437402cfe21ea22e460ac27b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/457036
Reviewed-by: Julie Qiu <julieqiu@google.com>
The default tab is changed to "doc".
The module path is now displayed on the module tab.
Updates b/124309981
Fixes b/131746914
Change-Id: I85957b1b7dd587ad54544e9c269f3f630619a32b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/457476
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
The following queries have been updated to fetch documentation from the
packages table:
* GetPackage
* GetLatestPackage
* GetVersionForPackage
The documentation tab now displays that data.
Fixes b/131706915
Change-Id: I591f374a355c55b8b937dbf2cfb2de6f105591cd
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456568
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
The string "TODO: <go documentation will go here>" is written to
packages.documentation from the fetch service.
This is an intermediary step to inserting the actual documentation.
Updates b/131706915
Change-Id: Id021e4cc3499f3727bcf561cdc3d23ecfc4a3311
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456566
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
This CL recreates the view created in
migrations/000023_add_explicit_license_sort.up.sql, to add the package
documentation column.
Updates b/131706915
Change-Id: Iefb300d66ad9d8fbbe9ecbc8e4d38b44008cc77b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456567
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
The search query is rewritten to:
* use documents.tsv_search_tokens instead of readme_tokens ||
synopsis_tokens || name_tokens || path_tokens
* query directly from documents, packages, versions, imports and
vw_package_licenses instead of using vw_search_results
* filter out results where relevance is < 1e-10 (instead of rank)
Based on my local database, which has 7185 packages, 144 versions and returns
800+ results for the search term "cloud", this reduced search time for "cloud"
from ~2.5secs to less than 300ms.
Updates b/130090305
Change-Id: Ida419e0d349b38149bec0e41db53b8bf0be18710
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456457
Reviewed-by: Robert Findley <rfindley@google.com>
Licenses were being sorted by Type, which need not be unique. Instead,
they should be sorted by FilePath.
Change-Id: I28205a6a0b075d8dd298b31846007009a24d3d18
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456233
Reviewed-by: Julie Qiu <julieqiu@google.com>
At the moment, there isn't a way to retry versions that have been
inserted into the version log if they are dropped after the
makeNewVersionsTimeout.
A /retry endpoint is added to the proxy index cron, which gets entries
from the version logs that are not present in the versions table and
have not failed, and retries fetching them.
The fetch service will now write errors to the error log.
This job is a temporary solution for populating the database, while the
go/go-discovery-etl design is being implemented.
Change-Id: I389c537068e05341ac79aac734d6df5766a1f278
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/452776
Reviewed-by: Robert Findley <rfindley@google.com>
This CL implements admittedly complicated logic for handling license
redistributability. It does this in three main steps, which could
conceivably be split into separate CLs if desirable.
1. Add an internal.Package.IsRedistributable method, which reports a
package as redistributable if and only if there is at least one license
permitting redistribution at the module root, and if every directory
containing a detected license file contains at least one license file
permitting redistribution. It is very possible that this logic is will
change in the future.
2. Add behavior in the postgres handler to prune package and version
data that should not be stored for non-redistributable content. As a
precaution it was decided to do this in the data layer rather than at
fetch time.
3. Add handling in the frontend to make it clear when content is not
redistributable.
Because we now have a scenario where we can't show license details, but
still want to surface the license disclaimer, move it to a separate
page.
Also:
+ modify license extraction to store the license path relative to the
module@version subdirectory of the module zip.
+ fix a bug where VersionType was not set on in postgres.GetPackage.
Fixes b/127335320
Updates b/124309095
Change-Id: I36194b78651ccff91b0e06d0d3d4fc639a884565
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/455069
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add an index.Client, analogous to the existing proxy.Client, that can be
used for fetching VersionLogs from the module index. Move the setup test
helper to index/test_helper.go. This client also implements a new
constraint to require https.
Also switch from reflect.DeepEqual to cmp.Diff for these tests, as it
produces a nice diff out of the box.
Updates b/130540724
Change-Id: I37542c84d8e68347e203e0ebaf81db924fb3b96e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456221
Reviewed-by: Julie Qiu <julieqiu@google.com>
Some modules haven't had a go.mod file added to the repository yet.
The go tool uses an implicit module with module path equal to the
import path corresponding to the repository root, and no other
directives.
When trying to process such a module as the main module with the
go/packages library, there needs to be an explicit go.mod file for
go/packages to be able to find where the module is on disk. So,
write the implicit go.mod file explicitly. It's guaranteed not to
exist (since we write it only when it doesn't exist).
Add a test case to TestExtractPackagesFromZip and update some error
messages in it.
Fixes b/130040000
Change-Id: I3eb211aa67b5d87122ce74eaf081f34350e061be
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456225
Reviewed-by: Julie Qiu <julieqiu@google.com>
Copy the cmd/go/internal/modfile package to internal/thirdparty.
It will be used in following commits to safely operate with go.mod
files.
Commands run to generate this change:
go run download.go -pkg=modfile
gofmt -r '"internal/lazyregexp" -> "regexp"' -w ./modfile/rule.go
gofmt -r 'lazyregexp.New -> regexp.MustCompile' -w ./modfile/rule.go
Change-Id: I44795395bd0dd4411c177b06fdbd95280afbd3a7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456224
Reviewed-by: Julie Qiu <julieqiu@google.com>
The search template has been updated to reflect the discovery site
alpha designs, leading to the following changes:
* SearchPage.NumPage has been added, which indicates the total number of
pages needed to display all of the search results (given the specified
limit)
* SearchPage.Total has been renamed to SearchPage.NumResults
* defaultSearchLimit = 10
Templates now support the functions "add" and "iterate".
Updates b/131593473
Change-Id: Ib57a8246199cee81682bb024ce9d317fde07e493
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/455285
Reviewed-by: Andrew Bonventre <andybons@google.com>
The navbar is updated and a footer is added to reflect the discovery
site alpha designs.
Updates b/131593473
Change-Id: I5d2d282e3b5eae6ae76e0d9c1cf216a61714fbe5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456216
Reviewed-by: Andrew Bonventre <andybons@google.com>
documents.path_tokens and documents.name_tokens will be removed for
tsv_search_tokens. These are set to not null for now while that
implementation is in progress.
Updates b/130090305
Change-Id: Ic95e5592aa0164bd3b5dd9b070fffc64ef2158d9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456223
Reviewed-by: Robert Findley <rfindley@google.com>
frontend.HandleDetails now uses module.CheckImportPath to check if the
<import-path> element is valid.
Previously, we were using module.CheckPath, which would fail on any path
ending in v1, such as google.golang.org/api/adexchangebuyer/v1.3.
Change-Id: Ifd5f57bdf9d5d1354def597fdeaa9329eab0f171
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456219
Reviewed-by: Robert Findley <rfindley@google.com>
The imports, importedby and module are updated to match the alpha designs.
Imports and ImportedBy now link to the latest version of each package and
display the package name.
A bug is fixed in HandleDetails, where an panic would occur if GetLatestPackage
returned an error and pkg.VersionInfo.Version was accessed immediately after.
Updates b/131593473
Change-Id: Icb42d1752f2d274cccc44cf5d47f994a3eaf2f62
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/455286
Reviewed-by: Robert Findley <rfindley@google.com>
Rather than storing the tsvector data for name, path, synopsis and readme
separately, they will now to be stored together in a tsv_search_tokens,
which has been shown to improve performance for the search query.
Change-Id: I80460b3a9e22efbe642419e37b95e10ed2c73d9f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/455288
Reviewed-by: Robert Findley <rfindley@google.com>
If a user makes a request to /<import-path>, without the <version> query
param, they will now be shown the latest version of the package.
Fixes b/130215822
Change-Id: Ia24c70ad720fd59f8cdf4dc21cded7beeb032bc5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/456217
Reviewed-by: Robert Findley <rfindley@google.com>
This change modifies the postgres test_helper API to make it easier to
use separate databases for each test suite. It also automates the test
database migration, to ensure that tests are run on an up-to-date
database schema.
Updates b/130719094
Change-Id: Iec0c2404b5859f9bb3c403bdec2581b4636f61a6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/454237
Reviewed-by: Julie Qiu <julieqiu@google.com>
Rob Findley
Julie Qiu
Dmitri Shuralyov
Andrew Bonventre