In the event that a fetched package is detected as not redistributable,
we will store only approved metadata. In this case, I think it is
helpful to distinguish records in the database that have only partial
data.
A use case is the following: suppose in the future there is a change to
the set of metadata we can store for non-redistributable packages. This
column will allow us to easily select those packages that need
reprocessing.
Note that we will always be looking at licenses on the fly to determine
if a package is redistributable. This column serves the purpose of
storing whether or not we considered the package redistributable at the
time we fetched it.
Change-Id: Ibc23b0c80919749bd6bee6ee8b2ba6a5c0c99c46
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/455068
Reviewed-by: Julie Qiu <julieqiu@google.com>
In these cases I assert that we don't need to distinguish between NULL
and empty string, so it is simpler to use NOT NULL columns than
sql.NullString.
This fixes some of the Scan() errors we have been encountering.
Fixes b/131325448
Change-Id: I7758dda89d6aa95408d04efdae9aa49dc2a2ffa9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/454617
Reviewed-by: Andrew Bonventre <andybons@google.com>
This allows callers to store and retrieve secrets stored in GCS and
encrypted using KMS. A simple API is exposed (Get and Set) and the
mechanisms used for storage and encryption are transparent to the
caller.
Initial support is added for the frontend binary. Some various
cleanup is also performed in that file.
Updates b/129370804
Change-Id: I8ebe08a8bf6c246b2b39a0a8d177147c32034758
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/454236
Reviewed-by: Robert Findley <rfindley@google.com>
All requests to the proxy should use valid TLS, so this CL modifies the
behavior of proxy.New to return an error if the proxy URL scheme is
anything other than https. This also involved updating the test setup to
use httptest.NewTLSServer.
Updates b/130540724
Change-Id: I87a98b20424cca59936df6115b285f0e5bc5acb0
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/453957
Reviewed-by: Julie Qiu <julieqiu@google.com>
To be consistent with other version information, use the version_type
stored on the versions table.
Updates b/130367504
Change-Id: I28940a30217f2abd1c688712ab89b5d4dce74d0e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/453885
Reviewed-by: Julie Qiu <julieqiu@google.com>
Since we're already joining with the versions table for other version
information, be consistent and store version_type as well. In subsequent
CLs, I will update the code to use this column, and drop unused version
information from the packages table.
Updates b/130367504
Change-Id: Id911108c8ca546fc40dfc7d33cd01d6de453ba57
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/453884
Reviewed-by: Julie Qiu <julieqiu@google.com>
Use module.CheckFilePath to guard against zipfile traversal, and add
tests for writeFileToDir.
In a subsequent CL, I will move functions related to module extraction
to unzip.go, so for now I have put tests in unzip_test.go.
Also, be consistent about calling filepath.FromSlash when converting the
zip file name to an os path.
Fixes b/130540930
Change-Id: Ic2d6f92a355e841bf26423c14f025eb41edaaef6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/453876
Reviewed-by: Julie Qiu <julieqiu@google.com>
staticcheck returned the following errors, which are fixed in this CL:
internal/frontend/details.go:227:6: func fetchPackageHeader is unused (U1000)
internal/frontend/details.go:513:2: redundant return statement (S1023)
internal/frontend/details_test.go:29:2: var samplePackageHeader is unused (U1000)
Change-Id: I35648d9e1934286c862d52823f6351d01afce81d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/453880
Reviewed-by: Robert Findley <rfindley@google.com>
The if check is not neccessary because the type assertion already
asserts that err is not nil. The behavior is unchanged without it.
This is a followup to CL 451521.
Change-Id: I403d091691c2d542859eac8ddd32c247a77cef37
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/453398
Reviewed-by: Robert Findley <rfindley@google.com>
This prevents execution from proceeding after the error gets handled.
This is a followup to CL 451521.
Change-Id: Ib121b04ed63a6362c19b2587569a0279dd2bd305
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/453397
Reviewed-by: Robert Findley <rfindley@google.com>
There are currently a non-trivial number of modules that are inserted
without packages, due to errors with packages.Load.
If a module does not have any packages, it will not show up on the
discovery site, so we should not insert that package to the database.
Updates b/130089785
Change-Id: I6608f737fcd362a40ca96f0e41f11be4b3abba4d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/452966
Reviewed-by: Robert Findley <rfindley@google.com>
To achieve this
+ refactor to lift up package fetch into the details handler.
+ update internal/postgres to return derrors.NotFound(...)
Updates b/127335320
Change-Id: I7a641472b514dcf09f08c3dea1e24a3a3a876713
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451521
Reviewed-by: Julie Qiu <julieqiu@google.com>
internal.Version has been updated with these fields:
* ReadmeFilePath: displays the file_path of the readme
* ReadmeContents: displays the contents of the readme, previously ReadMe
extractReadmeFromZip now returns both the file_path and contents of the
first README that is detected. The functions extractFile and
containsFile have been removed. Their functionality is now merged
inside extractReadmeFromZip.
readmeHTML now detects if a file is a markdown file, before parsing it
with the blackfriday markdown parser. If it is not, it returns the
readme as plaintext.
Several tests in internal/postgres have been updated to use
postgres.sampleVersions for generating test internal.Versions structs.
Change-Id: Ic0395d7c38090ace431c2ce9468d4ef5070bf2b4
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450586
Reviewed-by: Robert Findley <rfindley@google.com>
Break the reference cycle between 'Package' and 'Version', and get close
to the truth in the types returned from the database. To achieve this,
make the following changes:
+ Package type loses its Version reference
+ Version is split into metadata (VersionInfo) and Packages
+ VersionedPackage is introduced to capture the common case where we
want both Package and Version metadata.
Additionally, ensure that Suffix is always set on Packages returned from
the database.
There are still a few fields that can 'lie', notably Package imports and
Version readme, but those are left to a potential later CL.
Updates b/130367504
Change-Id: I392238aca4c335b64766c2ef23d145f462d0ff1b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451770
Reviewed-by: Julie Qiu <julieqiu@google.com>
The column versions.readme_contents is added, which will be used to
store contents of readme moving forward.
The contents of versions.readme are copied into
versions.readme_contents.
Once the code changes are made, versions.readme will be dropped.
Change-Id: Idd750a76b98e88a976f681b9c1ba3a3677c3bc2d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/452303
Reviewed-by: Robert Findley <rfindley@google.com>
The readme_file_path column is added to the versions table, which will be
used to determine the markup type that the readme is written in.
Change-Id: I2cfcd97a6e9e4bd102d1506916d5543e717e1927
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451769
Reviewed-by: Robert Findley <rfindley@google.com>
The query for postgres.Search is updated to use vw_search_results, so
that the search query is easier to read and maintain.
Results with a rank less than 1e-10 are filtered out.
The following fields for postgres.SearchResult are changed:
* Relevance is deprecated
* Rank is added, which is calculated using relevance and num imported by
* NumImportedBy is now used instead of NumImporters, for consistency with
the frontend.
* Total is added, which represents the total number of results from the query
postgres.GetLatestPackageForPaths is deprecated, since it is no longer used.
A bug is fixed in details.tmpl where the active tab in the nav bar was not
rendering for importedby.
Updates b/130090305
Change-Id: I8d72c730bd8667b941ca7a519c23234bd6bd646d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451331
Reviewed-by: Robert Findley <rfindley@google.com>
In many cases, these types were just holders for, respectively,
ModulePath and SeriesPath. Sometimes other fields were populated,
sometimes not. This change removes these types so that it is less
ambiguous what is available when working with our data models.
This also required joining with the modules table to get series_path in
some places. On principle I joined rather than ignore SeriesPath in
tests. We can later revisit simply making 'Series' a method on Version.
In postgres_test, I also introduced a pattern for generating test data
that I would like to consider for a more general test data library:
func sampleMyType(muts ...func(*MyType)) *MyType {...}
Updates b/130367504
Updates b/130719094
Change-Id: I66013fa79699bde9d09a37faef7928577163e0d2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451526
Reviewed-by: Julie Qiu <julieqiu@google.com>
The /favicon.ico route is handled explicity and no longer returns the
error: ParseModulePathAndVersion("/favicon.ico"): invalid path: "/favicon.ico"
Fixes b/129903914
Change-Id: I0044bf485d2d145e85d6ab2f90e82eece7305eab
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450682
Reviewed-by: Robert Findley <rfindley@google.com>
fetchImportersDetails now returns importers for a package.
importedby.tmpl is also updated to display the name and path of a
package's importers.
The tab importers is now importedby, for consistency with the rest of
the codebase.
Fixes b/124309193
Fixes b/126550318
Change-Id: I09f82d4a2b0a60e478e77268638cb7c3ac9cb75e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450579
Reviewed-by: Robert Findley <rfindley@google.com>
The search query is becoming difficult to read and maintain due to the
number of different tables that it involved.
A vw_search_results view is created, which contains all of the
information needed to generate search results.
postgres.Search is updated to use this view in CL 451331.
Updates b/130090305
Change-Id: I4149cd3e88e8b8b983578a56a9af294ea48d3af2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451330
Reviewed-by: Robert Findley <rfindley@google.com>
sos.googlesource.com/sos/license has been open sourced and now lives at
github.com/google/licensecheck. Use the open source license checker
instead of our private copy.
Fixes b/124309210
Change-Id: I3ac5d6c436dccca4d930d22df0d1162d89a0750f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451329
Reviewed-by: Robert Findley <rfindley@google.com>
This was an embarassing omission on my part, that I did not catch
because I was continually fetching packages for the first time. This
fixes the bug of duplicate licenses being associated with a package
after multiple fetches of the same version.
Updates b/129000846
Change-Id: I1e3eb594cbf73010dacf9d5d2f4237ce7463ff60
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/451313
Reviewed-by: Julie Qiu <julieqiu@google.com>
License detection seems like a good candidate to move to a separate
file, so I am making this change in advance of expanding the detection
logic.
This CL is purely moving code, and contains no change to the structure
of the program.
Updates b/130533761
Change-Id: I9a092cd331ce9d4f0e45881bdd20851c6ac7d62f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450857
Reviewed-by: Julie Qiu <julieqiu@google.com>
fetchImportsDetails now returns imports for a package from the
database. imports.tmpl is also updated to display the name and path of
imports, grouped by whether or not they are in the standard library.
A bug is fixed where the insert for to_name and to_path were
swapped.
Fixes b/124308480
Change-Id: Ic325aee1ee9be37aad7f13948d68ea85461dc92b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450576
Reviewed-by: Robert Findley <rfindley@google.com>
Add content to the licenses page, along with anchored links to specific
licenses that apply to the package, as well as a disclaimer.
Updates b/124309095
Change-Id: Id394c8acd2ca60cf711f64bc15541429e77715d1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450766
Reviewed-by: Julie Qiu <julieqiu@google.com>
Search results now show the number of packages that a given result is
imported by.
Updates b/130797141
Change-Id: I64dbaf66f7cbb09971c533ad72e77fc94aac6394
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450855
Reviewed-by: Robert Findley <rfindley@google.com>
There were a lot of duplicate testdata in details_test.go, which are now
consolidated into global variables to use across different tests.
Updates b/130719094
Change-Id: Ice046b3cede06a4653c1ffd7d9e43f7c83ab609a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450856
Reviewed-by: Robert Findley <rfindley@google.com>
Package imports are now parsed and inserted into
the database during the fetch process.
postgres.GetImports can be used to get imports for a given package from
the database.
Fixes b/126908435
Fixes b/125406053
Change-Id: I663a40ec89eb1fc4e4b76874aba4fcefafeddca0
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/437551
Reviewed-by: Andrew Bonventre <andybons@google.com>
frontend.Package now has the fields:
* Name: displays the package suffix as the name is main
* IsCommand: returns true if the package name is main.
* Title: displays Name, prefixed by "Command" or "Package"
These are generated inside createPackageHeader.
seriesPathForModule is deprecated in favor of module.SplitSemverPathVersion.
The content for helpers/_mod_header.tmpl and helpers/_modules_nav.tmpl
are moved to pages/details.tmpl. The package header now displays:
* "Command <name>" instead of "Package <name>" if IsCommand is true
* import "<path>" only if IsCommand is false
The links are versions.tmpl fixed to use the new v query param.
Fixes b/130320811
Change-Id: Ib4bb25ad54ff454e3b6b0a837f77a86dbfb60ccb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450575
Reviewed-by: Robert Findley <rfindley@google.com>
At least during the development phase, this will make it easier to
implement consistent styling across the site. If needed, we can later
either revert to constant values, or use a preprocessor.
Change-Id: I94d5987d69f2e251304c9c2907b0892c82c6ac76
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450849
Reviewed-by: Julie Qiu <julieqiu@google.com>
frontend.Package now has the methods:
* Name() string: displays the package suffix as the name is main
* IsCommand() bool: returns true if the package name is main.
* Title() string: displays Name(), prefixed by "Command" or "Package"
seriesPathForModule is deprecated in favor of module.SplitSemverPathVersion.
content/static/html/helpers/_mod_header.tmpl now displays:
* Package name using the Name() method
* "Command <name>" instead of "Package <name>" if IsCommand() is true
* import "<path>" only if IsCommand() is false
The links are versions.tmpl fixed to use the new v query param.
Fixes b/130320811
Change-Id: I32160d80df6312292c78007a84ac98867e9e7bf8
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/445774
Reviewed-by: Robert Findley <rfindley@google.com>
All of the handlers named fetch*Page (now renamed to fetch*Details)
return a PackageHeader and other details. They now all return a *DetailsPage,
with their corresponding details embedded, instead of different types.
The search handler logic is also moved to its own file at
internal/frontend/search.go
Fixes b/130721889
Change-Id: I5eca8cdda8fa658ec935f06521da2c8fe179b8d1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450569
Reviewed-by: Robert Findley <rfindley@google.com>
This method will be used to return full license contents for the
licenses details page.
Fixes b/129372226
Change-Id: I0efda0fad27328eec15a2a8fd1c65ebba1f78a5f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450455
Reviewed-by: Julie Qiu <julieqiu@google.com>
Eliminate usage of grpc.Status (which was just being used to communicate
InvalidArgument), by replacing it with the newly added derrors package.
Tests that were relying on grpc status codes for error assertions had to
be updated as well. I went through a few iterations of how to do this,
and settled on a simple enumeration of errors to be used for testing
only (in test_helper.go).
Also add documentation for postgres.DB and postgres.Open to make golint
happy.
Fixes b/128540225
Change-Id: Ie0b90211ae8d7d45daf980c1cc2701aabd4b9019
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450158
Reviewed-by: Julie Qiu <julieqiu@google.com>
The templates for the package details pages are refactored so that they
use a shared base layout.
Skeleton templates are also added for the imports and importers page.
Fixes b/130638124
Change-Id: I0e4072e91cd53ee61ff1a108592bcc2c4ed74235
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450209
Reviewed-by: Robert Findley <rfindley@google.com>
The route /favicon.ico is added, which returns a gopher as a favicon.
This also fixes a bug where each request to /<import-path> resulted in
two requests, since /favicon.io matches the syntax for /<import-path>.
Fixes b/130659698
Change-Id: I5aac0bea89832abfb57e70b3dd67441289e75d6b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/450210
Reviewed-by: Robert Findley <rfindley@google.com>
Add an internal errors package that implements some simple common error
semantics. This will be used to replace the usage of grpc/status.
Updates b/128540225
Change-Id: I43491178836af498c8e46afd99a6a649302cf2ae
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449575
Reviewed-by: Julie Qiu <julieqiu@google.com>
We are only storing direct imports for each package for now, so the
transitive column on the imports table is no longer needed.
Change-Id: I97be175c493c6e6a54d52429e7b904e4d4557cb7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449752
Reviewed-by: Robert Findley <rfindley@google.com>
Now that multi-license support has been added, make the
backward-incompatible schema change of dropping 'license' from the
versions table.
Also, add an explicit license sorting to the licenses that are returned
by the vw_licensed_packages query. Along the way, update a test to have
more distinct license (type, path) info.
Updates b/129372226
Change-Id: Ib66b6bb44f689a71ef0319526dd250be7e42e992
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449570
Reviewed-by: Julie Qiu <julieqiu@google.com>
postgres.bulkInsert is implemented, which allows for inserting multiple
rows in a single query.
Inserting multiple values with a single insert performed significantly
better than using a PREPARE statment, based on benchmarks in CL 449255.
postgres.versionsDiff is also removed from tests, and cmp.Diff is used instead.
Fixes b/129778736
Updates b/130092669
Change-Id: If38ae56859fa00cfb8ae2cb04b16d275db40f118
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/449253
Reviewed-by: Robert Findley <rfindley@google.com>
Rob Findley
Andrew Bonventre
Julie Qiu
Dmitri Shuralyov