This CL changes our data model so that licenses are associated with
packages rather than module versions, and adds multiple license
detection. As a result, given following file layout
my.mod/LICENSE
my.mod/pkgfoo/COPYING
my.mod/pkgbar/LICENSE.md
both LICENSE and COPYING (but not LICENSE.md) will be associated with
the package my.mod/pkgfoo.
Additionally make some minimal UI changes to support the new data model,
though a proper implementation of the licenses tab is deferred to a
later CL.
Fixes b/130372424
Updates b/129371814
Updates b/129372226
Fixes b/129000846
Fixes b/129372204
Change-Id: Ife58498914fea75e4c58e7c713a8071887f87000
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/447910
Reviewed-by: Julie Qiu <julieqiu@google.com>
There's no godoc content yet, that will follow in future CLs.
Change the tab name from "docs" to just "doc". This is simpler and more
consistent with "godoc", etc. The current future plan is to make this
the default tab, so it won't have a name then. But it'll continue to be
"doc" while the page is in development.
Change-Id: If4ec229a6b7f49aee216f6ea845bcf3d1e2fa356
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449355
Reviewed-by: Julie Qiu <julieqiu@google.com>
Additionally passes the template context to the nav template
on all pages with some other small updates.
Fixes b/130230126
Change-Id: I10114822382c6fc6c0a9f547749aa4faa433a851
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449352
Reviewed-by: Julie Qiu <julieqiu@google.com>
+ Disable autocomplete, spellcheck, autocorrect, etc. in the search
input
+ Remove redundant "equal" template function in favor of the built-in
"eq"
+ Populate the search input with the current query
+ Show the query first in the title since tabs are typically not
wide enough to show everything
+ Redirect to the homepage if the q param is empty
+ Other very small cleanup changes
Change-Id: Ib9baa41c336088cb3e9555ddaf527545292bb118
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449351
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add a licenses table to capture license information, along with a join
table that associates packages and licenses. This caused package
queries in internal/postgres to become pretty complicated, so add a
helper view to offload some of that complexity.
Fixes b/129370884
Change-Id: I7d97b3458b35c7c1b2400a28f5025ba3f83a5af1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/448258
Reviewed-by: Julie Qiu <julieqiu@google.com>
The fetchTimeout const was moved from cmd/cron/main.go to
internal/cron/cron.go in commit e51b998, but the signature of
cron.FetchVersions was not updated, which led to a compile error. This
CL fixes that bug.
Change-Id: I4ac4037e34115073e0ce96decbc696c4d9f74514
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/448153
Reviewed-by: Robert Findley <rfindley@google.com>
IntelliJ creates a .idea ignore in the repository root, which is now
ignored by git.
Change-Id: Ieda599bbc0865649e91bb4bee82b001d615a1026
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/447072
Reviewed-by: Robert Findley <rfindley@google.com>
The module page is added, which displays the directory and synopsis
of all packages in the module and the README.
This change includes the following additions:
* frontend.fetchModulePage: returns data needed to populate module.tmpl
* postgres.GetVersionForPackage: returns the version for a given
package_path and version, along with all of the packages for that
version.
Note: At the moment, both the overview page and the module page display
the README. Once a decision is made on where the README will be displayed,
these pages will be updated.
Fixes b/130112117
Fixes b/130111944
Change-Id: Ib240b1744cf66be72a63683804b521beaec62307
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444468
Reviewed-by: Andrew Bonventre <andybons@google.com>
Requests made from the cron endpoint /new/ to the fetch service are now
parallelized using cron.FetchVersions and by specifying a workers flag
for the maximum number of requests in flight.
Fixes b/128540164
Change-Id: Ic0885739ae668495cd91e5dacfd39846b16e83f3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/442300
Reviewed-by: Robert Findley <rfindley@google.com>
App Engine requires that apps listen on the port defined by the PORT
environment variable. Otherwise, it logs an error message like the
following:
App is listening on port 8080, it should instead listen on the port
defined by the PORT environment variable. As a consequence, nginx cannot
be started. Performance may be degraded. Please listen on the port
defined by the PORT environment variable.
Change-Id: I38f8e0e96cc44e384fb712510efbfaaa68fef4f4
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/446246
Reviewed-by: Robert Findley <rfindley@google.com>
extractPackagesFromZip is updated so that it will skip non-.go files
greater than 10MB when parsing packages. If there is a .go file in the
module greater than 10MB, an error will be returned.
detectLicense is updated so that it will skip license files found
that are greater than 10MB.
Fixes b/130089430
Change-Id: I8ea97d514b7b07968bbc219c9b257f125df50e33
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/446245
Reviewed-by: Robert Findley <rfindley@google.com>
This CL implements a pattern similar to that used in
github.com/golang/gddo to implement an active tab selector for the nav.
Along the way, refactor a bit so that the frontend package is fully
responsible for templates.
Also replace the pattern of Make*HandlerFunc with handler methods.
Also fix a minor typo in all.bash that was causing misspell not to be
installed.
Updates b/124309095
Change-Id: I802ba69b91b81be13c6314140326fe8783608a0c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/445973
Reviewed-by: Julie Qiu <julieqiu@google.com>
This fixes a latent bug where clock offsets cause us to miss versions.
Spanner provides a monotonicity guarantee so that its commit time is
viable for pagination, if not ideal.
Unfortunately it's only monotonic: querying for since=t_0 returns all
versions with Timestamp >= t_0, and therefore we'll always have an
at-least-1 module overlap in our polling.
Fixes b/129700670
Change-Id: Iab9a2ed0445c913ef5f1d590d30152adda9a157e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444659
Reviewed-by: Julie Qiu <julieqiu@google.com>
The GO_DISCOVERY_[service]_ADDR variable is introduced, to set the adrr
that various services are running on. The default is set to
localhost:[port] to mute to popup on incoming networking connections
when running locally.
The default for GO_MODULE_PROXY_URL is set to https://proxy.golang.org
and for GO_MODULE_PROXY_INDEX is set to https://index.golang.org/index.
Change-Id: I097fd96a5c51ce1ec73ceab1928fda39de506352
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/445646
Reviewed-by: Robert Findley <rfindley@google.com>
Previously, an error occurred when parsing versions with build tags,
such as "v1.2.5+incompatible".
This CL fixes the error and moves parsing major, minor and patch inside
internal/postgres to individual functions.
Fixes#130041347.
Change-Id: I192dad8b1620c2235f19ddc8cf0f04f98c473b39
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444465
Reviewed-by: Andrew Bonventre <andybons@google.com>
The links in versions.tmpl now route to the default tab instead of the
versions tabs.
An error with using the wrong type to format an arg for the
createPackageHeader error message is also fixed.
Change-Id: I5479895e13e04867aa00153e4276e0e6fc4aee43
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444466
Reviewed-by: Robert Findley <rfindley@google.com>
The following endpoints now have handler funcs HTML/CSS:
- "/" directs the user to the home page which contains just a search bar.
This endpoint is handled by MakeDetailsHandlerFunc.
- "/search?q=<query>" directs the user to a list of search results for the
specified query. This endpint is handled by MakeSearchHandlerFunc.
Fixes b/125408009.
Change-Id: I2d919e35b8d5d8d5f76a6895d47f99897b4f32ff
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444660
Reviewed-by: Julie Qiu <julieqiu@google.com>
validateVersions used to return an error if a version did not have any
packages. These versions should be inserted into the database, since we
could have a module without any packages.
Error messages corresponding to a given version no longer print the full
struct. They will only print the module path and version.
GetLatestPackageForPaths is also moved to internal/postgres/search.go, since
it is used for search.
Fixes b/130027196.
Change-Id: Ief0ee0ab977a47ee0840da14ef2336f082b2db28
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444661
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Use the Timeout middleware to to add a timeout for incoming requests.
As a notable exception, don't do this for module fetch, as we want
module processing to continue even if the client cancels their request
early. Instead, set an explicit timeout for module processing.
Additionally, bind work done on behalf of the request to the request
context. This required switching to the context-aware sql APIs, setting
a context on outbound HTTP requests, and quite a bit of 'ctx' threading.
Fixes b/128689909
Change-Id: Ic732808b6df89a61106f5d14b8a9ecaaefb95c4f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443023
Reviewed-by: Julie Qiu <julieqiu@google.com>
When a new version is downloaded, insert it into the
documents table so that it can be used for search.
Change-Id: Id46bc0aa6b14f0c5bc9f7520aef3baad8efe0454
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444658
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
extractPackagesFromZip opened and closed files in a loop by calling
defer file.Close(). This led to an error with too many files being open
when unzipping large modules.
The files are now closed immediately after use, using the func
writeFileToDir.
Fixes b/129349898.
Change-Id: If71ab136920ee2c51e20d4c501c2cfd0d7962f22
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444549
Reviewed-by: Robert Findley <rfindley@google.com>
This CL addresses two causes of flaky tests:
+ Non-deterministic sorting in search results, due to results of the
same rank.
+ Transactional issues due to multiple non-hermetic package tests
running in parallel.
Fixes b/129855505
Fixes b/129963750
Fixes b/129696857
Change-Id: Ic69cb1f6393325f4c3955d721a4dbdcf73d8258d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444670
Reviewed-by: Julie Qiu <julieqiu@google.com>
fetchVersionsTabPage takes in a package path and a version and then returns
all of the data needed to display the packages that have the same series and
package suffix to make the versions tab content. The data is displayed in a
hierarchical list of major (i.e. "v1") then major and minor (i.e. "1.2") and
lastly the entirety of the version (i.e. "1.2.3" or "1.2.2-alpha.1").
verisons.tmpl is the template that displays all of the data in the
VersionsTabPage struct.
The routing was changed so that MakeFrontendHandlerFunc handles both
/<import path>@<version> which brings up the overview page and
/<import path>@<version>?tab=versions which brings up the versions
tab.
Fixes b/124308701
Change-Id: I92bd86d050ab6e7e669c07a750b325b3e026d052
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443954
Reviewed-by: Julie Qiu <julieqiu@google.com>
frontend.fetchSearchPage is implemented, which returns all of the data
needed to generate results for a search page.
postgres.Search is updated so that results with a rank score of 0 are
not returned.
Change-Id: I2e41716ce0cfa9db70b0a84930b4e2568a2c969b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443906
Reviewed-by: Robert Findley <rfindley@google.com>
Truncate all timestamps used in tests to Microsecond precision, so that
precision is not lost when saving and retrieving from Postgres.
Fixes b/129859073
Change-Id: Id677147ac44dc08cd52e0fae9a6d0510b68a666e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443892
Reviewed-by: Andrew Bonventre <andybons@google.com>
InsertDocuments inserts the packages for a given version into the
documents table, and creates tsvector tokens to allow for searching
package name, package path, synopsis tokens, and readme tokens.
Search fetches packages that match a given set of terms, and returns
them in order of rank. Rank is based on relevance and the number of
dependents.
Change-Id: I60e8b690955adb66306e326238e426cf77527f82
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443363
Reviewed-by: Andrew Bonventre <andybons@google.com>
This adds a simple definition of middleware, along with a single
implementation: a Timeout middleware. In a follow-up CL I'll use this
to add timeouts to user-facing handlers.
Updates b/128689909
Change-Id: Ie2078e4e3da45767c7ef38f1aab25b8da5b1cf14
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443949
Reviewed-by: Andrew Bonventre <andybons@google.com>
Depending on locale settings, it is possible for '~' to be ordered as
less than 'a'. This breaks the semver sorting logic. To prevent this,
set explicit "C" collation for versions.prerelease, packages.path,
and packages.module_path.
Fixes b/129684816
Change-Id: I6e290d7d1e55c2979f5c6dcee7a5c00fdecdd680
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443305
Reviewed-by: Julie Qiu <julieqiu@google.com>
This CL edits GetTaggedVersions and GetPseudoVersions so that
they fetch all of the versions of associated packages that
have the same specified series name and package path suffix.
These methods will be used to determine which versions to
display on the versions tab in the frontend.
Fixes b/129553236
Change-Id: I3930868b66f11d955f1ce15307aa21c0599ab999
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443090
Reviewed-by: Julie Qiu <julieqiu@google.com>
GetLatestPackageForPaths originally was not pulling expected packages,
due to how the WHERE clause param was being passed in.
It is fixed in this CLs along with the tests.
Change-Id: Ia377175bd6a51fe4fd53ea3d8ccc2c1dfc377ecb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443360
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Remove the K8s-applied fields from service.yaml so that we can run
`kubectl apply -f` on this resource instead of running
`kubectl expose`.
Change-Id: I1399658df53d73d6021979a1334503d8e1510876
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443102
Reviewed-by: Megan O'Keefe <mokeefe@google.com>
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
This CL adds error checking for the semver parsing in InsertVersion.
It also fixes patch parsing so that it properly isolates the number
string when there is a prerelease.
Fixes b/129717943
Change-Id: I542b533d9508027917b1f9fa63867607b31d23fc
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443170
Reviewed-by: Julie Qiu <julieqiu@google.com>
cmd/fetch/main.go can now be deployed to kubernetes following the
instructions in cmd/fetch/README.md, using the Dockerfile and
deployment.yaml in cmd/fetch.
Change-Id: I22c22ccc40059f999ce3373e4f0a8530feaba3b7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/440415
Reviewed-by: Andrew Bonventre <andybons@google.com>
This CL parses the package path in order to fill the 'suffix'
column in the packages table. There is currently no need to
get the value back from the database, it will be used to
fetch all the versions of a particular package in a series.
Fixes b/129546260
Change-Id: I403220f0d3ad90be4c30b563f1fe38d86a2f31df
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/442297
Reviewed-by: Julie Qiu <julieqiu@google.com>
GetTaggedVersions and GetPseudoVersions both return a list of
versions for a specific package. These methods will be used
by the frontend to fetch the versions that will be displayed
in the 'versions' tab.
Fixes b/126550849
Change-Id: I169aad79823d67273cb7ab1f3995b700d75e8ac2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/441783
Reviewed-by: Julie Qiu <julieqiu@google.com>
In order to eventually display all the package versions associated
with a specific series this new package suffix column will allow us
to join the 'packages' and 'modules' tables and fetch all the package
versions with the same package suffix and series name. These versions
will then be listed on the versions tab for the frontend.
Fixes b/129545571
Change-Id: I790187a0ea56cd238c7f88603995ea0fe883ba95
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/442289
Reviewed-by: Julie Qiu <julieqiu@google.com>
The module overview page is being updated to be a package
overview page instead. The package header information
has been pulled out into its own struct because that information
will need to be used across various tabs and GetPackage was
updated to also return the module path and readme from the
database.
Fixes b/129020591
Change-Id: I09174c68077b3dc9c27da608b219cabf610639ee
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/441334
Reviewed-by: Julie Qiu <julieqiu@google.com>
The proxy index field "Name" is now "Path.
The JSON struct tag for VersionLog.ModulePath is updated to reflect that change.
Change-Id: I4a6488e2c55a9ba6e3fd6943551c2e997b61b42f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/440424
Reviewed-by: Andrew Bonventre <andybons@google.com>
stmt.Close() is called to free any resources used to create a prepared
statement for inserting packages inside postgres.InsertVersion.
Change-Id: I84ca3128af503b343f0b7f811090ddd8cb9226ab
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/440425
Reviewed-by: Andrew Bonventre <andybons@google.com>
InsertVersion now checks to make sure that the module path is valid
before inserting a version into the database.
parseModulePathAndVersion now checks the module path and returns an
error if the path is invalid.
FetchAndInsertVersion also ensures the module path is valid before
calling InsertVersion.
Fixes b/129269917
Change-Id: I1f9446a671b9af29e48b7cc5554007fda657662a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/440419
Reviewed-by: Julie Qiu <julieqiu@google.com>
This change adds GetLatestPackageForPaths which returns the latest
package associated with each path in a list of strings. This method
will eventually be used to get the packages queried by search results.
Updates b/124308701
Fixes b/126714352
Change-Id: I2b6e86aec7b5ba82ed0e45db52784d3cde1b8074
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/439699
Reviewed-by: Julie Qiu <julieqiu@google.com>
Rob Findley
Dmitri Shuralyov
Andrew Bonventre
Julie Qiu
Channing Kimble-Brown