InsertDocuments inserts the packages for a given version into the
documents table, and creates tsvector tokens to allow for searching
package name, package path, synopsis tokens, and readme tokens.
Search fetches packages that match a given set of terms, and returns
them in order of rank. Rank is based on relevance and the number of
dependents.
Change-Id: I60e8b690955adb66306e326238e426cf77527f82
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443363
Reviewed-by: Andrew Bonventre <andybons@google.com>
This adds a simple definition of middleware, along with a single
implementation: a Timeout middleware. In a follow-up CL I'll use this
to add timeouts to user-facing handlers.
Updates b/128689909
Change-Id: Ie2078e4e3da45767c7ef38f1aab25b8da5b1cf14
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443949
Reviewed-by: Andrew Bonventre <andybons@google.com>
This CL edits GetTaggedVersions and GetPseudoVersions so that
they fetch all of the versions of associated packages that
have the same specified series name and package path suffix.
These methods will be used to determine which versions to
display on the versions tab in the frontend.
Fixes b/129553236
Change-Id: I3930868b66f11d955f1ce15307aa21c0599ab999
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443090
Reviewed-by: Julie Qiu <julieqiu@google.com>
GetLatestPackageForPaths originally was not pulling expected packages,
due to how the WHERE clause param was being passed in.
It is fixed in this CLs along with the tests.
Change-Id: Ia377175bd6a51fe4fd53ea3d8ccc2c1dfc377ecb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443360
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
This CL adds error checking for the semver parsing in InsertVersion.
It also fixes patch parsing so that it properly isolates the number
string when there is a prerelease.
Fixes b/129717943
Change-Id: I542b533d9508027917b1f9fa63867607b31d23fc
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/443170
Reviewed-by: Julie Qiu <julieqiu@google.com>
This CL parses the package path in order to fill the 'suffix'
column in the packages table. There is currently no need to
get the value back from the database, it will be used to
fetch all the versions of a particular package in a series.
Fixes b/129546260
Change-Id: I403220f0d3ad90be4c30b563f1fe38d86a2f31df
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/442297
Reviewed-by: Julie Qiu <julieqiu@google.com>
GetTaggedVersions and GetPseudoVersions both return a list of
versions for a specific package. These methods will be used
by the frontend to fetch the versions that will be displayed
in the 'versions' tab.
Fixes b/126550849
Change-Id: I169aad79823d67273cb7ab1f3995b700d75e8ac2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/441783
Reviewed-by: Julie Qiu <julieqiu@google.com>
The module overview page is being updated to be a package
overview page instead. The package header information
has been pulled out into its own struct because that information
will need to be used across various tabs and GetPackage was
updated to also return the module path and readme from the
database.
Fixes b/129020591
Change-Id: I09174c68077b3dc9c27da608b219cabf610639ee
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/441334
Reviewed-by: Julie Qiu <julieqiu@google.com>
The proxy index field "Name" is now "Path.
The JSON struct tag for VersionLog.ModulePath is updated to reflect that change.
Change-Id: I4a6488e2c55a9ba6e3fd6943551c2e997b61b42f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/440424
Reviewed-by: Andrew Bonventre <andybons@google.com>
stmt.Close() is called to free any resources used to create a prepared
statement for inserting packages inside postgres.InsertVersion.
Change-Id: I84ca3128af503b343f0b7f811090ddd8cb9226ab
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/440425
Reviewed-by: Andrew Bonventre <andybons@google.com>
InsertVersion now checks to make sure that the module path is valid
before inserting a version into the database.
parseModulePathAndVersion now checks the module path and returns an
error if the path is invalid.
FetchAndInsertVersion also ensures the module path is valid before
calling InsertVersion.
Fixes b/129269917
Change-Id: I1f9446a671b9af29e48b7cc5554007fda657662a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/440419
Reviewed-by: Julie Qiu <julieqiu@google.com>
This change adds GetLatestPackageForPaths which returns the latest
package associated with each path in a list of strings. This method
will eventually be used to get the packages queried by search results.
Updates b/124308701
Fixes b/126714352
Change-Id: I2b6e86aec7b5ba82ed0e45db52784d3cde1b8074
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/439699
Reviewed-by: Julie Qiu <julieqiu@google.com>
The proxy requires the <module> and <version> elements to be case-encoded
in requests. The proxy client is updated to encode these elements using
internal/thirdparty/module.EncodePath.
Change-Id: Ie5d6b6ab4156efa8a4d6ac36dbe9b0ef743f3d9a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/439140
Reviewed-by: Andrew Bonventre <andybons@google.com>
This CL parses the version for the semver columns in
InsertVersion so that we can query the database for packages sorted by
semver.
This also adds the GetLatestPackage which will be used by the frontend
when a user specifies the package path but not the version.
Fixes b/129020053
Change-Id: I8741721f61aa78f0e7dca358345903488df8a6fa
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/438011
Reviewed-by: Julie Qiu <julieqiu@google.com>
+ Fix a test that fails due to precision differences in time values
in Postgres
+ Provide an example URL when visiting the root path of the fetch
service
+ Update handlers to return a 404 instead of 500 if a user provides
an invalid path
+ Don't require a full URL when parsing module data since the path
is all that's needed
Change-Id: I9f8dc13b0a4995e6cb848f86ad41f6fed2759f82
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/439693
Reviewed-by: Julie Qiu <julieqiu@google.com>
The package cmd/go/internal/module is cloned to
internal/thirdparty/module for use in this project, using the cmd `go
run internal/thirdparty/download.go -pkg=module`
It will be used to encode module path and version in requests to the proxy.
Change-Id: Icb81ea6dd46b1a5826cc8586a5b358057d81d066
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/439709
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
This change updates the parseVersion so that it returns both a
VersionType and an error and adds a VersionType field to the
Version struct.
Change-Id: I35527ab135b21502beead3103d5075914da49a37
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437555
Reviewed-by: Julie Qiu <julieqiu@google.com>
READMEs written in markdown will be displayed as HTML.
internal.Version.ReadMe is changed from type string to []byte.
Minor changes are made to the Overview page to reflect changes for
package discovery.
Change-Id: I52e5ae091cf8069154415e02c7d88c7911f3a205
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437553
Reviewed-by: Andrew Bonventre <andybons@google.com>
The url format for the details page is changed to
/<path>@<version>?tab=<tab> from /<path>?v=<version>?tab=<tab>.
The <version> param is also now validated using
internal/thirdparty/semver.IsValid to ensure it is a valid semantic
version.
Change-Id: Iae8384ab41a94a4e4f19d7129bda531d9966543e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437552
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Reviewed-by: Andrew Bonventre <andybons@google.com>
The database needs an enum column called version_type to indicate
the type of version a package has. So to be able to insert
packages with the correct version type VersionType is being added
along with function getVersionType to parse a package's
version to determine what kind of VersionType it should have.
getVersionType is also used to make sure that the version is valid
in FetchAndInsertVersion.
Change-Id: Ia15c23c3783dbb83934ce8b0b580a25bafa219c2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437333
Reviewed-by: Andrew Bonventre <andybons@google.com>
The package cmd/go/internal/semver is cloned to internal/thirdparty/semver
for use in this project, using the cmd
`go run internal/thirdparty/download.go -pkg=semver`
Change-Id: Ic569891486469768dcdd1ad79f33c6e80b1ccda8
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437550
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
A bug introduced in commit cdaa29e is fixed. The bug was due to
Module.Name being renamed to Module.Path throughout the codebase, but
not in the templates inside content/.
Change-Id: Iadf2d85e0f33c2dfb57c2ff1f88e7914d9d6e5d7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437549
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
When a module zip is downloaded by the fetch service, its packages will
be extracted. The synopsis for each package will also be determined
using go/doc.
These packages will be written to postgres with postgres.InsertVersion.
Because we expected to add more version data over time, so
postgres.InsertVersion will no longer fail on duplicate key errors.
A package can be retrieved with postgres.GetPackage.
Change-Id: I9916d82c1479914f0b91ca02105aab86e578aac7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/436915
Reviewed-by: Andrew Bonventre <andybons@google.com>
Zip files are now ignored by git, since the binary files created are
different depending on the machine.
Change-Id: I1d21482f9d8b6cfee6c8cf60c5419da260527ba0
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/436919
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
This CL updates zipfiles that should have been modified in commit
00942cf.
Change-Id: I6b55885d5b575a95adda3286c3ff1d9f1961ee3e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/436914
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
This CL fixes a bug that was introduced in 00942cf:
internal/proxy/test_helper.go:65:2: missing return at end of function
Change-Id: I0be608aabaf7ddd8ead91b7dc898275a0d54fc73
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/436187
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
cmd/go refers to "module name" as "module path". References to module
and series name in the discovery codebase are edited to follow that
naming convention.
Change-Id: Ia2c7335d10f172dfd7a6c30eda0b30d1a7a5525d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/435779
Reviewed-by: Andrew Bonventre <andybons@google.com>
All module zips inside internal/proxy/testdata were previously created
manually. They are now created automatically when tests are run.
Testdata for modules that were previoulsy inside internal/fetch/testdata
are moved to internal/proxy/testdata.
Tests inside internal/proxy and internal/fetch are updated to reflect
this changes.
Change-Id: I348fd4ebfe6c4ec0751945966984afc3859a2330
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/435777
Reviewed-by: Andrew Bonventre <andybons@google.com>
The following build errors are fixed:
introduced in commit 9c415db174e
cmd/cron/main.go:42:16: undefined: cron.NewVersionsFromProxyIndex
introduced in commit 08ffd9f83
internal/frontend/handlers_test.go:159:3: Fatalf format %q has arg tc.version of wrong type golang.org/x/discovery/internal.Version
Change-Id: Iecafaccb48a4a3b5b6c720d8272c9ff683f082d4
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/435029
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
The proxy index cron is created, with a job to get new versions from the
module index. The cron will:
1. query the module index for new versions since a given timestamp
2. write each version to the version_logs table
3. make a request to the fetch service for each version to be
downloaded
The fetch client is also fixed to make a GET request to
https://<fetch-url>/<module>/@v/<version>.
Change-Id: I838029d94f9b2782e0c1066ec7932931b47fe01e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/426749
Reviewed-by: Andrew Bonventre <andybons@google.com>
There is now a frontend binary that accepts HTTP requests for:
GET /name?v=<version>
This route renders an overview page, which displays the name, version
publish date and readme for a specified module. The readme has not yet
been rendered into HTML.
Also note that the tabs are not fully functional or accessible and
these issues will be thoroughly addressed in a future CL.
Fixes b/124438879
Fixes b/124438779
Fixes b/124438684
Change-Id: I569dd3549bdea781183e2d7d29be375141fb5c3d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/425454
Reviewed-by: Andrew Bonventre <andybons@google.com>
The command internal/thirdparty/download.go is added, which is used to
download cmd/go/internal packages from https://go.googlesource.com/go.
It alsos fixes the imports
The cmd/go/internal packages module and semver are added using this
tool. These will be used to get requirements from a given mod file.
Change-Id: I81b0e065df0d7f544dcb6e5c985082a61866d7c9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/424991
Reviewed-by: Andrew Bonventre <andybons@google.com>
detectLicenses is implemented, which searches for possible license files
in the contents directory of the provided zip path, runs them against a
license classifier, and provides all licenses with a confidence score
that are above 97%.
Change-Id: I91333beb9a84f1a89a7bbd054c1b81c98022d1ce
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/426792
Reviewed-by: Katie Hockman <katiehockman@google.com>
The error message for attempting to insert an invalid version is updated
to specify the fields that cause the error.
Change-Id: Ic92da9940c6fa337d082eb126e25aee42591721d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/426796
Reviewed-by: Katie Hockman <katiehockman@google.com>
- GetVersion now reads from the 'license' and 'readme' columns.
- InsertVersion now also tests to make sure that none of the required fields
are nil or empty strings before insertion and returns an error indicating
what is wrong with the input.
- Also added status codes to the InsertVersion errors to make
testing easier.
Fixes b/124339314
Fixes b/124340579
Change-Id: I37c969bf4618a00cf220f99ecb2ff088bd5af009
Reviewed-on: https://team-review.git.corp.google.com/c/423000
Reviewed-by: Katie Hockman <katiehockman@google.com>
A fetch service is created, which accepts HTTP requests for:
GET /<module>/@v/<version>
This route invokes internal/fetch.FetchAndInsertVersion, which will:
(1) Downloads the given module version from the module proxy
(2) Process the contents of th zip.
(3) Writes the data to postgres.
extractPackagesFromZip is also fixed to handle an error with os.Create,
where the directory of the file being written didn't already exist.
Change-Id: I7b9d4c8941eadc2bc71286715f764de2ad471aa1
Reviewed-on: https://team-review.git.corp.google.com/c/424990
Reviewed-by: Andrew Bonventre <andybons@google.com>
An initial version FetchAndInsertVersion is implemented, which:
(1) Downloads the given module version from the module proxy
(2) Process the contents:
- Calculate series name
- Get the contents of the README
- Get the contents of the license
- Get packages for the module
(3) Writes the data to postgres.
Change-Id: Iabce23879124a03599dc5779302e69cc5e432f88
Reviewed-on: https://team-review.git.corp.google.com/c/422995
Reviewed-by: Andrew Bonventre <andybons@google.com>
The fetch service accepts HTTP requests at
http(s)://<fetchURL>/<module>@<version>.
ParseNameAndVersion is implemented, which validates the module name and
version for requests to this endpoint.
Change-Id: Iacf7f4ae8dfef54a024172807d8bc9ff01749c84
Reviewed-on: https://team-review.git.corp.google.com/c/422997
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Reviewed-by: Andrew Bonventre <andybons@google.com>
The Version struct is updated to reflect that change from the migration in
092e728bbbfdd15a40165f52110a6aec4bf55548:
- Version.LicenseName is renamed to License
- Version.LicenseContents is removed
SQL queries for postgres.GetVersion and postgres.InsertVersion are
also fixed to reflect the migration changes.
Change-Id: I3a9c7cc9e02ca839741acac7a8370c4c6237fe7e
Reviewed-on: https://team-review.git.corp.google.com/c/423370
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
The ReadMe and License structs in internal/discovery.go are being merged
into the Version struct to reflect the corresponding tables being
merged into the versions table in the database.
Change-Id: I78a2c05733bb935140c9c4c86237e673921f5b36
Reviewed-on: https://team-review.git.corp.google.com/c/422452
Reviewed-by: Julie Qiu <julieqiu@google.com>
packagesInModuleZip is implemented, which parses the zip for a module version and returns
its packages.
Change-Id: I5fb12dfbe3b5912fa748ce5b77cd47f01055d60a
Reviewed-on: https://team-review.git.corp.google.com/c/420373
Reviewed-by: Andrew Bonventre <andybons@google.com>
The series name will be the shared base path of a group of major-version
variants. For example, my/module, my/module/v2, my/module/v3 are a
single series, with the series name my/module.
Change-Id: Ia984c572f4cc04836a187e04b81279e0f756f5ba
Reviewed-on: https://team-review.git.corp.google.com/c/419129
Reviewed-by: Andrew Bonventre <andybons@google.com>
GetVersion and InsertVersion are implemented. GetVersion takes
a primary key, the name and version, and returns the corresponding
Version if it exists in the Versions table. InsertVersion takes
a Version and inserts it into the Versions table if there are no
pre-existing Versions with the same primary key, failing otherwise.
GetVersion particularly will be used to get the information needed
to write the html header for the discovery site.
Note that the updated_at field in the Versions schema should use a trigger
to automatically update but that will be handled in a future CL.
Fixes b/124338357
Change-Id: If6ecc43a35381814f74df024581a135f16c34771
Reviewed-on: https://team-review.git.corp.google.com/c/417750
Reviewed-by: Katie Hockman <katiehockman@google.com>
GetZip is implemented, which makes a request to
$GOPROXY/<module>/@v/<version>.zip and transforms that data into a
*zip.Reader
This will be used by the discovery fetch service to download modules
from the proxy.
Change-Id: I5aacbbc12fd01f77a22a415206e64698d8849d40
Reviewed-on: https://team-review.git.corp.google.com/c/417749
Reviewed-by: Andrew Bonventre <andybons@google.com>
The fetch service accepts POST requests at /<module>@<version> for
downloading versions and writing them to the database.
This implements a client for making this request.
Change-Id: Ib39e1e8393eda6e79bc32abaeeed1839c5a235cf
Reviewed-on: https://team-review.git.corp.google.com/c/417466
Reviewed-by: Andrew Bonventre <andybons@google.com>
This fixes the build error for cleanURL from
It also moves discovery/internal/proxyclient to discovery/internal/proxy
and renames the package proxyclient to proxy.
Change-Id: I8ea20164dbe24b3c15601b6a5eabe750db3b48a0
Reviewed-on: https://team-review.git.corp.google.com/c/418403
Reviewed-by: Andrew Bonventre <andybons@google.com>
The discovery/internal/proxyclient package is created to interact with a
module proxy. See `go help goproxy` for proxy protocol details.
The following structs are introduced:
- proxyclient.Client
- proxyclient.VersionInfo
The following exported functions are implemented:
- New(rawurl string) (*Client, error)
- GetInfo(name, version string) (*VersionInfo, error)
discovery/internal/testdata is moved to discovery/internal/fetch/testdata.
Change-Id: I3b0fa8dcf95f8dbbc39519640adbf199c1e61efb
Reviewed-on: https://team-review.git.corp.google.com/c/414334
Reviewed-by: Andrew Bonventre <andybons@google.com>
The fetch service will eventually read the README from a zip that is
downloaded from the module proxy. The README will be written to the
database and displayed on the discovery site.
Change-Id: I9c028497dcff50e0b0edda8c8483cd824e3abcc4
Reviewed-on: https://team-review.git.corp.google.com/c/415623
Reviewed-by: Andrew Bonventre <andybons@google.com>
The discovery/internal/sourcestorage package is created to read/write
from a GCS bucket.
The following methods are implemented:
- OpenBucket(ctx context.Context, bucket string) (*Bucket, error)
- Write(ctx context.Context, key string, data []byte) error
- Read(ctx context.Context, key string) ([]byte, error)
Change-Id: Ib85f4beba32b08c3cd9935ba5792de22dd76ed45
Reviewed-on: https://team-review.git.corp.google.com/c/414335
Reviewed-by: Andrew Bonventre <andybons@google.com>
The discovery/internal/postgres package is created to interact
with the discovery database. The following methods are implemented:
- LastestProxyIndexUpdate() *time.Time
- InsertNewVersionsToLog(newVersions []*internal.VersionLog) error
These will be used by the proxy index cron when fetching new versions
from the module index.
The version_log table is renamed to version_logs. This was initially
a typo.
A dependency to github.com/lib/pq v1.0.0 is introduced.
Change-Id: I720a836dc85a37a5df863e879ce6f60082c795f1
Reviewed-on: https://team-review.git.corp.google.com/c/413847
Reviewed-by: Andrew Bonventre <andybons@google.com>
Implemented a proxy that runs on localhost:8080 following the Module
proxy protocol.
It supports the modules and versions:
- my/module (v1.0.0, v1.1.0, v1.1.1)
- my/module/v2 (v2.0.0)
A GET request can be made to:
- GET $GOPROXY/<module>/@v/list
- GET $GOPROXY/<module>/@v/<version>.info
- GET $GOPROXY/<module>/@v/<version>.mod
- GET $GOPROXY/<module>/@v/<version>.zip
All the zip files contain only an empty README.md.
Change-Id: If7d547eb94a9a24608fafccdfd39330da9849a0e
Reviewed-on: https://team-review.git.corp.google.com/c/412883
Reviewed-by: Andrew Bonventre <andybons@google.com>
Julie Qiu
Rob Findley
Channing Kimble-Brown
Andrew Bonventre