This change modifies the postgres test_helper API to make it easier to
use separate databases for each test suite. It also automates the test
database migration, to ensure that tests are run on an up-to-date
database schema.
Updates b/130719094
Change-Id: Iec0c2404b5859f9bb3c403bdec2581b4636f61a6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/454237
Reviewed-by: Julie Qiu <julieqiu@google.com>
fetchImportersDetails now returns importers for a package.
importedby.tmpl is also updated to display the name and path of a
package's importers.
The tab importers is now importedby, for consistency with the rest of
the codebase.
Fixes b/124309193
Fixes b/126550318
Change-Id: I09f82d4a2b0a60e478e77268638cb7c3ac9cb75e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450579
Reviewed-by: Robert Findley <rfindley@google.com>
sos.googlesource.com/sos/license has been open sourced and now lives at
github.com/google/licensecheck. Use the open source license checker
instead of our private copy.
Fixes b/124309210
Change-Id: I3ac5d6c436dccca4d930d22df0d1162d89a0750f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451329
Reviewed-by: Robert Findley <rfindley@google.com>
fetchImportsDetails now returns imports for a package from the
database. imports.tmpl is also updated to display the name and path of
imports, grouped by whether or not they are in the standard library.
A bug is fixed where the insert for to_name and to_path were
swapped.
Fixes b/124308480
Change-Id: Ic325aee1ee9be37aad7f13948d68ea85461dc92b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450576
Reviewed-by: Robert Findley <rfindley@google.com>
Add content to the licenses page, along with anchored links to specific
licenses that apply to the package, as well as a disclaimer.
Updates b/124309095
Change-Id: Id394c8acd2ca60cf711f64bc15541429e77715d1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450766
Reviewed-by: Julie Qiu <julieqiu@google.com>
All of the handlers named fetch*Page (now renamed to fetch*Details)
return a PackageHeader and other details. They now all return a *DetailsPage,
with their corresponding details embedded, instead of different types.
The search handler logic is also moved to its own file at
internal/frontend/search.go
Fixes b/130721889
Change-Id: I5eca8cdda8fa658ec935f06521da2c8fe179b8d1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450569
Reviewed-by: Robert Findley <rfindley@google.com>
Eliminate usage of grpc.Status (which was just being used to communicate
InvalidArgument), by replacing it with the newly added derrors package.
Tests that were relying on grpc status codes for error assertions had to
be updated as well. I went through a few iterations of how to do this,
and settled on a simple enumeration of errors to be used for testing
only (in test_helper.go).
Also add documentation for postgres.DB and postgres.Open to make golint
happy.
Fixes b/128540225
Change-Id: Ie0b90211ae8d7d45daf980c1cc2701aabd4b9019
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450158
Reviewed-by: Julie Qiu <julieqiu@google.com>
postgres.bulkInsert is implemented, which allows for inserting multiple
rows in a single query.
Inserting multiple values with a single insert performed significantly
better than using a PREPARE statment, based on benchmarks in CL 449255.
postgres.versionsDiff is also removed from tests, and cmp.Diff is used instead.
Fixes b/129778736
Updates b/130092669
Change-Id: If38ae56859fa00cfb8ae2cb04b16d275db40f118
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449253
Reviewed-by: Robert Findley <rfindley@google.com>
This CL changes our data model so that licenses are associated with
packages rather than module versions, and adds multiple license
detection. As a result, given following file layout
my.mod/LICENSE
my.mod/pkgfoo/COPYING
my.mod/pkgbar/LICENSE.md
both LICENSE and COPYING (but not LICENSE.md) will be associated with
the package my.mod/pkgfoo.
Additionally make some minimal UI changes to support the new data model,
though a proper implementation of the licenses tab is deferred to a
later CL.
Fixes b/130372424
Updates b/129371814
Updates b/129372226
Fixes b/129000846
Fixes b/129372204
Change-Id: Ife58498914fea75e4c58e7c713a8071887f87000
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/447910
Reviewed-by: Julie Qiu <julieqiu@google.com>
There's no godoc content yet, that will follow in future CLs.
Change the tab name from "docs" to just "doc". This is simpler and more
consistent with "godoc", etc. The current future plan is to make this
the default tab, so it won't have a name then. But it'll continue to be
"doc" while the page is in development.
Change-Id: If4ec229a6b7f49aee216f6ea845bcf3d1e2fa356
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449355
Reviewed-by: Julie Qiu <julieqiu@google.com>
+ Disable autocomplete, spellcheck, autocorrect, etc. in the search
input
+ Remove redundant "equal" template function in favor of the built-in
"eq"
+ Populate the search input with the current query
+ Show the query first in the title since tabs are typically not
wide enough to show everything
+ Redirect to the homepage if the q param is empty
+ Other very small cleanup changes
Change-Id: Ib9baa41c336088cb3e9555ddaf527545292bb118
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449351
Reviewed-by: Julie Qiu <julieqiu@google.com>
Requests made from the cron endpoint /new/ to the fetch service are now
parallelized using cron.FetchVersions and by specifying a workers flag
for the maximum number of requests in flight.
Fixes b/128540164
Change-Id: Ic0885739ae668495cd91e5dacfd39846b16e83f3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/442300
Reviewed-by: Robert Findley <rfindley@google.com>
extractPackagesFromZip is updated so that it will skip non-.go files
greater than 10MB when parsing packages. If there is a .go file in the
module greater than 10MB, an error will be returned.
detectLicense is updated so that it will skip license files found
that are greater than 10MB.
Fixes b/130089430
Change-Id: I8ea97d514b7b07968bbc219c9b257f125df50e33
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/446245
Reviewed-by: Robert Findley <rfindley@google.com>
This fixes a latent bug where clock offsets cause us to miss versions.
Spanner provides a monotonicity guarantee so that its commit time is
viable for pagination, if not ideal.
Unfortunately it's only monotonic: querying for since=t_0 returns all
versions with Timestamp >= t_0, and therefore we'll always have an
at-least-1 module overlap in our polling.
Fixes b/129700670
Change-Id: Iab9a2ed0445c913ef5f1d590d30152adda9a157e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444659
Reviewed-by: Julie Qiu <julieqiu@google.com>
validateVersions used to return an error if a version did not have any
packages. These versions should be inserted into the database, since we
could have a module without any packages.
Error messages corresponding to a given version no longer print the full
struct. They will only print the module path and version.
GetLatestPackageForPaths is also moved to internal/postgres/search.go, since
it is used for search.
Fixes b/130027196.
Change-Id: Ief0ee0ab977a47ee0840da14ef2336f082b2db28
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/444661
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Use the Timeout middleware to to add a timeout for incoming requests.
As a notable exception, don't do this for module fetch, as we want
module processing to continue even if the client cancels their request
early. Instead, set an explicit timeout for module processing.
Additionally, bind work done on behalf of the request to the request
context. This required switching to the context-aware sql APIs, setting
a context on outbound HTTP requests, and quite a bit of 'ctx' threading.
Fixes b/128689909
Change-Id: Ic732808b6df89a61106f5d14b8a9ecaaefb95c4f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443023
Reviewed-by: Julie Qiu <julieqiu@google.com>
frontend.fetchSearchPage is implemented, which returns all of the data
needed to generate results for a search page.
postgres.Search is updated so that results with a rank score of 0 are
not returned.
Change-Id: I2e41716ce0cfa9db70b0a84930b4e2568a2c969b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443906
Reviewed-by: Robert Findley <rfindley@google.com>
+ Fix a test that fails due to precision differences in time values
in Postgres
+ Provide an example URL when visiting the root path of the fetch
service
+ Update handlers to return a 404 instead of 500 if a user provides
an invalid path
+ Don't require a full URL when parsing module data since the path
is all that's needed
Change-Id: I9f8dc13b0a4995e6cb848f86ad41f6fed2759f82
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/439693
Reviewed-by: Julie Qiu <julieqiu@google.com>
This change updates the parseVersion so that it returns both a
VersionType and an error and adds a VersionType field to the
Version struct.
Change-Id: I35527ab135b21502beead3103d5075914da49a37
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437555
Reviewed-by: Julie Qiu <julieqiu@google.com>
READMEs written in markdown will be displayed as HTML.
internal.Version.ReadMe is changed from type string to []byte.
Minor changes are made to the Overview page to reflect changes for
package discovery.
Change-Id: I52e5ae091cf8069154415e02c7d88c7911f3a205
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437553
Reviewed-by: Andrew Bonventre <andybons@google.com>
The database needs an enum column called version_type to indicate
the type of version a package has. So to be able to insert
packages with the correct version type VersionType is being added
along with function getVersionType to parse a package's
version to determine what kind of VersionType it should have.
getVersionType is also used to make sure that the version is valid
in FetchAndInsertVersion.
Change-Id: Ia15c23c3783dbb83934ce8b0b580a25bafa219c2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437333
Reviewed-by: Andrew Bonventre <andybons@google.com>
When a module zip is downloaded by the fetch service, its packages will
be extracted. The synopsis for each package will also be determined
using go/doc.
These packages will be written to postgres with postgres.InsertVersion.
Because we expected to add more version data over time, so
postgres.InsertVersion will no longer fail on duplicate key errors.
A package can be retrieved with postgres.GetPackage.
Change-Id: I9916d82c1479914f0b91ca02105aab86e578aac7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/436915
Reviewed-by: Andrew Bonventre <andybons@google.com>
All module zips inside internal/proxy/testdata were previously created
manually. They are now created automatically when tests are run.
Testdata for modules that were previoulsy inside internal/fetch/testdata
are moved to internal/proxy/testdata.
Tests inside internal/proxy and internal/fetch are updated to reflect
this changes.
Change-Id: I348fd4ebfe6c4ec0751945966984afc3859a2330
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/435777
Reviewed-by: Andrew Bonventre <andybons@google.com>
The replace directive for sos.googlesource.com/sos is moved to
the root of the discovery repo to fix a build error when deploying GAE.
Fixes b/128783757.
Change-Id: I02d290f2f98e161e992b8191d9f74a8fc2d54531
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/435776
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Some of the migrations files were missing licenses or said
'Copyright 2009' instead of 'Copyright 2019' and needed to be
fixed.
Change-Id: I36a6c627b8893ad227b608ea96d4f257e9ab0562
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/435003
Reviewed-by: Julie Qiu <julieqiu@google.com>
The proxy index cron is created, with a job to get new versions from the
module index. The cron will:
1. query the module index for new versions since a given timestamp
2. write each version to the version_logs table
3. make a request to the fetch service for each version to be
downloaded
The fetch client is also fixed to make a GET request to
https://<fetch-url>/<module>/@v/<version>.
Change-Id: I838029d94f9b2782e0c1066ec7932931b47fe01e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/426749
Reviewed-by: Andrew Bonventre <andybons@google.com>
There is now a frontend binary that accepts HTTP requests for:
GET /name?v=<version>
This route renders an overview page, which displays the name, version
publish date and readme for a specified module. The readme has not yet
been rendered into HTML.
Also note that the tabs are not fully functional or accessible and
these issues will be thoroughly addressed in a future CL.
Fixes b/124438879
Fixes b/124438779
Fixes b/124438684
Change-Id: I569dd3549bdea781183e2d7d29be375141fb5c3d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/425454
Reviewed-by: Andrew Bonventre <andybons@google.com>
The command internal/thirdparty/download.go is added, which is used to
download cmd/go/internal packages from https://go.googlesource.com/go.
It alsos fixes the imports
The cmd/go/internal packages module and semver are added using this
tool. These will be used to get requirements from a given mod file.
Change-Id: I81b0e065df0d7f544dcb6e5c985082a61866d7c9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/424991
Reviewed-by: Andrew Bonventre <andybons@google.com>
detectLicenses is implemented, which searches for possible license files
in the contents directory of the provided zip path, runs them against a
license classifier, and provides all licenses with a confidence score
that are above 97%.
Change-Id: I91333beb9a84f1a89a7bbd054c1b81c98022d1ce
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/426792
Reviewed-by: Katie Hockman <katiehockman@google.com>
The error message for attempting to insert an invalid version is updated
to specify the fields that cause the error.
Change-Id: Ic92da9940c6fa337d082eb126e25aee42591721d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/426796
Reviewed-by: Katie Hockman <katiehockman@google.com>
An initial version FetchAndInsertVersion is implemented, which:
(1) Downloads the given module version from the module proxy
(2) Process the contents:
- Calculate series name
- Get the contents of the README
- Get the contents of the license
- Get packages for the module
(3) Writes the data to postgres.
Change-Id: Iabce23879124a03599dc5779302e69cc5e432f88
Reviewed-on: https://team-review.git.corp.google.com/c/422995
Reviewed-by: Andrew Bonventre <andybons@google.com>
packagesInModuleZip is implemented, which parses the zip for a module version and returns
its packages.
Change-Id: I5fb12dfbe3b5912fa748ce5b77cd47f01055d60a
Reviewed-on: https://team-review.git.corp.google.com/c/420373
Reviewed-by: Andrew Bonventre <andybons@google.com>
The discovery/internal/sourcestorage package is created to read/write
from a GCS bucket.
The following methods are implemented:
- OpenBucket(ctx context.Context, bucket string) (*Bucket, error)
- Write(ctx context.Context, key string, data []byte) error
- Read(ctx context.Context, key string) ([]byte, error)
Change-Id: Ib85f4beba32b08c3cd9935ba5792de22dd76ed45
Reviewed-on: https://team-review.git.corp.google.com/c/414335
Reviewed-by: Andrew Bonventre <andybons@google.com>
The discovery/internal/postgres package is created to interact
with the discovery database. The following methods are implemented:
- LastestProxyIndexUpdate() *time.Time
- InsertNewVersionsToLog(newVersions []*internal.VersionLog) error
These will be used by the proxy index cron when fetching new versions
from the module index.
The version_log table is renamed to version_logs. This was initially
a typo.
A dependency to github.com/lib/pq v1.0.0 is introduced.
Change-Id: I720a836dc85a37a5df863e879ce6f60082c795f1
Reviewed-on: https://team-review.git.corp.google.com/c/413847
Reviewed-by: Andrew Bonventre <andybons@google.com>
Rob Findley
Julie Qiu
Dmitri Shuralyov
Andrew Bonventre
Channing Kimble-Brown