The replace directive for sos.googlesource.com/sos is moved to
the root of the discovery repo to fix a build error when deploying GAE.
Fixes b/128783757.
Change-Id: I02d290f2f98e161e992b8191d9f74a8fc2d54531
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/435776
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
The NOT NULL constraints on versions.major, versions.minor and
versions.patch are resulting in some tests failures because the
ingestion logic is not yet implemented.
The constraints are dropped for now and may be added back later.
Fixes b/128831211.
Change-Id: I7f2dec952f6ce7c39bf344dfdd45bedebb375e20
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/435778
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Some of the migrations files were missing licenses or said
'Copyright 2009' instead of 'Copyright 2019' and needed to be
fixed.
Change-Id: I36a6c627b8893ad227b608ea96d4f257e9ab0562
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/435003
Reviewed-by: Julie Qiu <julieqiu@google.com>
The following build errors are fixed:
introduced in commit 9c415db174e
cmd/cron/main.go:42:16: undefined: cron.NewVersionsFromProxyIndex
introduced in commit 08ffd9f83
internal/frontend/handlers_test.go:159:3: Fatalf format %q has arg tc.version of wrong type golang.org/x/discovery/internal.Version
Change-Id: Iecafaccb48a4a3b5b6c720d8272c9ff683f082d4
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/435029
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Major, minor, patch, prerelease and build columns are being added to
allow for sorting queries by semantic version.
Updates b/126714352
Change-Id: I791cb38b69e5661e682d4d06fb37d0d20fe0a7a7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/434769
Reviewed-by: Andrew Bonventre <andybons@google.com>
The proxy index cron is created, with a job to get new versions from the
module index. The cron will:
1. query the module index for new versions since a given timestamp
2. write each version to the version_logs table
3. make a request to the fetch service for each version to be
downloaded
The fetch client is also fixed to make a GET request to
https://<fetch-url>/<module>/@v/<version>.
Change-Id: I838029d94f9b2782e0c1066ec7932931b47fe01e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/426749
Reviewed-by: Andrew Bonventre <andybons@google.com>
There is now a frontend binary that accepts HTTP requests for:
GET /name?v=<version>
This route renders an overview page, which displays the name, version
publish date and readme for a specified module. The readme has not yet
been rendered into HTML.
Also note that the tabs are not fully functional or accessible and
these issues will be thoroughly addressed in a future CL.
Fixes b/124438879
Fixes b/124438779
Fixes b/124438684
Change-Id: I569dd3549bdea781183e2d7d29be375141fb5c3d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/425454
Reviewed-by: Andrew Bonventre <andybons@google.com>
The command internal/thirdparty/download.go is added, which is used to
download cmd/go/internal packages from https://go.googlesource.com/go.
It alsos fixes the imports
The cmd/go/internal packages module and semver are added using this
tool. These will be used to get requirements from a given mod file.
Change-Id: I81b0e065df0d7f544dcb6e5c985082a61866d7c9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/424991
Reviewed-by: Andrew Bonventre <andybons@google.com>
detectLicenses is implemented, which searches for possible license files
in the contents directory of the provided zip path, runs them against a
license classifier, and provides all licenses with a confidence score
that are above 97%.
Change-Id: I91333beb9a84f1a89a7bbd054c1b81c98022d1ce
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/426792
Reviewed-by: Katie Hockman <katiehockman@google.com>
The error message for attempting to insert an invalid version is updated
to specify the fields that cause the error.
Change-Id: Ic92da9940c6fa337d082eb126e25aee42591721d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/426796
Reviewed-by: Katie Hockman <katiehockman@google.com>
Versions without a license will be inserted as a blacklist, so that
users can't make requests to keep triggering another fetch.
Versions.license will as a result sometimes be NULL.
Change-Id: If35b822ced2bc2792d2959ea5af1716fa473a366
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/426800
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
- GetVersion now reads from the 'license' and 'readme' columns.
- InsertVersion now also tests to make sure that none of the required fields
are nil or empty strings before insertion and returns an error indicating
what is wrong with the input.
- Also added status codes to the InsertVersion errors to make
testing easier.
Fixes b/124339314
Fixes b/124340579
Change-Id: I37c969bf4618a00cf220f99ecb2ff088bd5af009
Reviewed-on: https://team-review.git.corp.google.com/c/423000
Reviewed-by: Katie Hockman <katiehockman@google.com>
A fetch service is created, which accepts HTTP requests for:
GET /<module>/@v/<version>
This route invokes internal/fetch.FetchAndInsertVersion, which will:
(1) Downloads the given module version from the module proxy
(2) Process the contents of th zip.
(3) Writes the data to postgres.
extractPackagesFromZip is also fixed to handle an error with os.Create,
where the directory of the file being written didn't already exist.
Change-Id: I7b9d4c8941eadc2bc71286715f764de2ad471aa1
Reviewed-on: https://team-review.git.corp.google.com/c/424990
Reviewed-by: Andrew Bonventre <andybons@google.com>
An initial version FetchAndInsertVersion is implemented, which:
(1) Downloads the given module version from the module proxy
(2) Process the contents:
- Calculate series name
- Get the contents of the README
- Get the contents of the license
- Get packages for the module
(3) Writes the data to postgres.
Change-Id: Iabce23879124a03599dc5779302e69cc5e432f88
Reviewed-on: https://team-review.git.corp.google.com/c/422995
Reviewed-by: Andrew Bonventre <andybons@google.com>
The fetch service accepts HTTP requests at
http(s)://<fetchURL>/<module>@<version>.
ParseNameAndVersion is implemented, which validates the module name and
version for requests to this endpoint.
Change-Id: Iacf7f4ae8dfef54a024172807d8bc9ff01749c84
Reviewed-on: https://team-review.git.corp.google.com/c/422997
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Reviewed-by: Andrew Bonventre <andybons@google.com>
The Version struct is updated to reflect that change from the migration in
092e728bbbfdd15a40165f52110a6aec4bf55548:
- Version.LicenseName is renamed to License
- Version.LicenseContents is removed
SQL queries for postgres.GetVersion and postgres.InsertVersion are
also fixed to reflect the migration changes.
Change-Id: I3a9c7cc9e02ca839741acac7a8370c4c6237fe7e
Reviewed-on: https://team-review.git.corp.google.com/c/423370
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Since the license_contents column is being removed we can simplify
the column name 'license_name' to just 'license'. The contents of
the licenses can be accessed using GCS links so there's no need to
store the contents in the database.
Change-Id: I9df43ec3d098eacc78ffddd55fe3e8147ea6615c
Reviewed-on: https://team-review.git.corp.google.com/c/423369
Reviewed-by: Julie Qiu <julieqiu@google.com>
The ReadMe and License structs in internal/discovery.go are being merged
into the Version struct to reflect the corresponding tables being
merged into the versions table in the database.
Change-Id: I78a2c05733bb935140c9c4c86237e673921f5b36
Reviewed-on: https://team-review.git.corp.google.com/c/422452
Reviewed-by: Julie Qiu <julieqiu@google.com>
Removing the license column in the database schema and adding the NOT NULL
constraint to the license_name column. This is all part of an effort to
merge the ReadMe and License structs into the Version struct so that
Version will reflect the new flattened Versions table in the database.
Change-Id: If24943492d6700719622c0537504137d12fa322f
Reviewed-on: https://team-review.git.corp.google.com/c/422453
Reviewed-by: Julie Qiu <julieqiu@google.com>
packagesInModuleZip is implemented, which parses the zip for a module version and returns
its packages.
Change-Id: I5fb12dfbe3b5912fa748ce5b77cd47f01055d60a
Reviewed-on: https://team-review.git.corp.google.com/c/420373
Reviewed-by: Andrew Bonventre <andybons@google.com>
Currently, the tables readmes and licenses have a 1 to 1 relationship
with versions. The only reason that data for these tables are separate
is because they are expected to be very large. However, Postgres will
automatically handle large values:
"Long strings are compressed by the system automatically, so the
physical requirement on disk might be less. Very long values are also
stored in background tables so that they do not interfere with rapid
access to shorter column values."
(https://www.postgresql.org/docs/9.6/datatype-character.html)
Therefore, the columns from readmes and licenses are moved to the
versions table.
License headers are also added to migrations 000003, 000004, and 000005.
Change-Id: I3aff7f8eb83ea22b48c2081a853e5cd9b161319d
Reviewed-on: https://team-review.git.corp.google.com/c/421150
Reviewed-by: Channing Kimble-Brown <ckimblebrown@google.com>
Reviewed-by: Andrew Bonventre <andybons@google.com>
The series name will be the shared base path of a group of major-version
variants. For example, my/module, my/module/v2, my/module/v3 are a
single series, with the series name my/module.
Change-Id: Ia984c572f4cc04836a187e04b81279e0f756f5ba
Reviewed-on: https://team-review.git.corp.google.com/c/419129
Reviewed-by: Andrew Bonventre <andybons@google.com>
GetVersion and InsertVersion are implemented. GetVersion takes
a primary key, the name and version, and returns the corresponding
Version if it exists in the Versions table. InsertVersion takes
a Version and inserts it into the Versions table if there are no
pre-existing Versions with the same primary key, failing otherwise.
GetVersion particularly will be used to get the information needed
to write the html header for the discovery site.
Note that the updated_at field in the Versions schema should use a trigger
to automatically update but that will be handled in a future CL.
Fixes b/124338357
Change-Id: If6ecc43a35381814f74df024581a135f16c34771
Reviewed-on: https://team-review.git.corp.google.com/c/417750
Reviewed-by: Katie Hockman <katiehockman@google.com>
Creates a migration that sets the deleted field in the versions
schema to false by default.
Fixes b/125424519
Change-Id: I5740841e2f6e147120b3d9514d57266f08fc94b7
Reviewed-on: https://team-review.git.corp.google.com/c/420912
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add a trigger that updates the 'updated_at' field in the Versions
table upon any update and make the initial default value of 'updated_at'
the time of transaction.
Change-Id: I1d82a7e064e4cef71b8d61adf045a5131abe417f
Reviewed-on: https://team-review.git.corp.google.com/c/420309
Reviewed-by: Julie Qiu <julieqiu@google.com>
The created_at columns of series, versions, modules, and version_logs
are given a default timestamp of current_timestamp.
A created_at column is also added to the documents table, with the same
default timestamp of current_timestamp.
Change-Id: I93960de99f7c785ece4d3028ea1cbdfaa54a94a9
Reviewed-on: https://team-review.git.corp.google.com/c/419132
Reviewed-by: Andrew Bonventre <andybons@google.com>
GetZip is implemented, which makes a request to
$GOPROXY/<module>/@v/<version>.zip and transforms that data into a
*zip.Reader
This will be used by the discovery fetch service to download modules
from the proxy.
Change-Id: I5aacbbc12fd01f77a22a415206e64698d8849d40
Reviewed-on: https://team-review.git.corp.google.com/c/417749
Reviewed-by: Andrew Bonventre <andybons@google.com>
The fetch service accepts POST requests at /<module>@<version> for
downloading versions and writing them to the database.
This implements a client for making this request.
Change-Id: Ib39e1e8393eda6e79bc32abaeeed1839c5a235cf
Reviewed-on: https://team-review.git.corp.google.com/c/417466
Reviewed-by: Andrew Bonventre <andybons@google.com>
This fixes the build error for cleanURL from
It also moves discovery/internal/proxyclient to discovery/internal/proxy
and renames the package proxyclient to proxy.
Change-Id: I8ea20164dbe24b3c15601b6a5eabe750db3b48a0
Reviewed-on: https://team-review.git.corp.google.com/c/418403
Reviewed-by: Andrew Bonventre <andybons@google.com>
The LICENSE file formatting is updated to match the standard Go BSD
license.
Change-Id: I507d7d13e01873cfac6ab8f359de4f56409b4497
Reviewed-on: https://team-review.git.corp.google.com/c/417742
Reviewed-by: Andrew Bonventre <andybons@google.com>
The discovery/internal/proxyclient package is created to interact with a
module proxy. See `go help goproxy` for proxy protocol details.
The following structs are introduced:
- proxyclient.Client
- proxyclient.VersionInfo
The following exported functions are implemented:
- New(rawurl string) (*Client, error)
- GetInfo(name, version string) (*VersionInfo, error)
discovery/internal/testdata is moved to discovery/internal/fetch/testdata.
Change-Id: I3b0fa8dcf95f8dbbc39519640adbf199c1e61efb
Reviewed-on: https://team-review.git.corp.google.com/c/414334
Reviewed-by: Andrew Bonventre <andybons@google.com>
The fetch service will eventually read the README from a zip that is
downloaded from the module proxy. The README will be written to the
database and displayed on the discovery site.
Change-Id: I9c028497dcff50e0b0edda8c8483cd824e3abcc4
Reviewed-on: https://team-review.git.corp.google.com/c/415623
Reviewed-by: Andrew Bonventre <andybons@google.com>
The discovery/internal/sourcestorage package is created to read/write
from a GCS bucket.
The following methods are implemented:
- OpenBucket(ctx context.Context, bucket string) (*Bucket, error)
- Write(ctx context.Context, key string, data []byte) error
- Read(ctx context.Context, key string) ([]byte, error)
Change-Id: Ib85f4beba32b08c3cd9935ba5792de22dd76ed45
Reviewed-on: https://team-review.git.corp.google.com/c/414335
Reviewed-by: Andrew Bonventre <andybons@google.com>
The discovery/internal/postgres package is created to interact
with the discovery database. The following methods are implemented:
- LastestProxyIndexUpdate() *time.Time
- InsertNewVersionsToLog(newVersions []*internal.VersionLog) error
These will be used by the proxy index cron when fetching new versions
from the module index.
The version_log table is renamed to version_logs. This was initially
a typo.
A dependency to github.com/lib/pq v1.0.0 is introduced.
Change-Id: I720a836dc85a37a5df863e879ce6f60082c795f1
Reviewed-on: https://team-review.git.corp.google.com/c/413847
Reviewed-by: Andrew Bonventre <andybons@google.com>
Implemented a proxy that runs on localhost:8080 following the Module
proxy protocol.
It supports the modules and versions:
- my/module (v1.0.0, v1.1.0, v1.1.1)
- my/module/v2 (v2.0.0)
A GET request can be made to:
- GET $GOPROXY/<module>/@v/list
- GET $GOPROXY/<module>/@v/<version>.info
- GET $GOPROXY/<module>/@v/<version>.mod
- GET $GOPROXY/<module>/@v/<version>.zip
All the zip files contain only an empty README.md.
Change-Id: If7d547eb94a9a24608fafccdfd39330da9849a0e
Reviewed-on: https://team-review.git.corp.google.com/c/412883
Reviewed-by: Andrew Bonventre <andybons@google.com>
Julie Qiu
Channing Kimble-Brown
Andrew Bonventre