Add a mode to postgres.DB where data is saved even if the module or
package is not redistributable. This "bypassing license restrictions"
mode is off by default.
Add a flag to the worker binary to turn it on. Used a flag rather than
an environment variable so there is no way for ambient state to affect
the binary for this important decision.
Also:
- Add a test to verify that license data is removed/not removed
depending on the bypass setting.
- Fix a bug where the new model insertion wasn't omitting
documentation for non-redistributable modules.
- Fix a bug in internal/sample where the top-level readme wasn't
getting populated.
A later CL will change the frontend to bypass the check.
For golang/go#39602
For golang/go#39629
Change-Id: I67a6d24c18f3b93cfbfc9ec2a20159c07a84e077
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/246957
Reviewed-by: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
The New suffix is removed from PackageNew and DirectoryNew, since the
legacy methods are already prefixed with "Legacy".
For golang/go#39629
Change-Id: I8be32310c6b451035021663000aac6ca845ee6e0
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241900
Reviewed-by: Jonathan Amsterdam <jba@google.com>
GetPackagesInDirectory is added, which uses the paths table to get all
of the packages within a given directory path. This will replace
LegacyGetDirectory and LegacyGetPackagesInModule in creating the
directories tab.
For golang/go#39629
Change-Id: I4b39690e838756421bee9b5ca389081e184761bf
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/241838
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Change the type of the documentation fields from string to safehtml.HTML.
This forces the conversions between strings and safehtml.HTML values
to happen at the lowest level, reading and writing to the database.
That increases our confidence that nothing will modify the HTML after
it is read from the DB and before it is used in a template.
Change-Id: If5d65123ce2b69ef3183221048aa6d081b003762
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240510
Reviewed-by: Julie Qiu <julie@golang.org>
GitHub paths typically have three components. Change our sample module
path to match that URL pattern.
Change-Id: I049df016b46affd942209a2aee798fdb5537e0a6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/240943
Reviewed-by: Jonathan Amsterdam <jba@google.com>
postgres.DeleteModule is updated to delete the corresponding module_path
and resolved_version row from the version_map table.
Fixes#39633
Change-Id: I9de46b08e535bea52c2ea4dbab1c71a8e1c2c2f7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238637
Reviewed-by: Julie Qiu <julie@golang.org>
DirectoryMeta is added, which contains metadata for a directory.
This struct will be used to return the list of directories in a given
path in a follow up CL.
Change-Id: I89880bd8104791510bc07c1fb3532f6a71bed24c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/238245
Reviewed-by: Jonathan Amsterdam <jba@google.com>
ModuleInfo is added, which represents the module info for the new data
model. ModuleInfo is embedded in LegacyModuleInfo.
In order to support the existing overview tab functionality,
GetDirectoryNew was changed to return the module README, regardless of
whether there is a README for the directory.
We will change this logic to display the README for the directory in a
future CL.
Change-Id: I624a6d99b711870826fd7dff9100d4ad47852db2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766801
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When we insert module information, make sure we overwrite existing
rows. Previously, we specified that new data should be ignored if the
row already existed, which meant that we were adding rows but never
changing them. That didn't matter at one point, when we deleted a
module before re-inserting it, but we no longer delete, so we must
upsert.
We also fix a number of places where our test modules had duplicate
directories. The upserts failed for these.
Change-Id: I97465b11e4ea6cbb7835e883f36f098445207eba
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766365
Reviewed-by: Julie Qiu <julieqiu@google.com>
ModuleInfo is renamed to LegacyModuleInfo, as a step towards
deprecating LegacyReadmeFilePath and LegacyReadmeContents.
In a follow up CL, we will add ModuleInfo as an embedded struct
to LegacyModuleInfo.
Change-Id: Ie452420448eec1d13edaf62e548df0b9e2cbbe4b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766479
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The following structs have been renamed with a Legacy prefix:
* internal.Directory
* internal.Package
* internal.VersionedPackage
* sample.Package
The following fields on internal.ModuleInfo have also been changed:
* ReadmeFilePath
* ReadmeFileContents
This is done to help us distinguish between legacy and method
structs/methods while migrating code to the new data model.
Change-Id: Ibedf71d4db6323ef5aa05d73a0240537ea6073d3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/765160
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Construct the sample SourceInfo from the given module path, not
the global one.
Change-Id: I2d61d0b234c591a2a6fb03bfbf329549cc0df507
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750122
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Use GetPathInfo to determine how to serve a path.
Protect behind an experiment.
Change-Id: Ia78b0049e4d76216654216c66a6c0786f38bd8f5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/731801
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Replace many calls to sample.DefaultModule, and all calls to
sample.DefaultPackage, with calls to the new Module and Package
functions.
Change-Id: I76921e14502585f8ca9a4dba5de01d7055522f3f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/733604
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Redo the arguments to Package so it can derive the Name
and V1Path fields from them.
- Add a list of suffixes to the arguments of Module.
Change-Id: I966197846b49befceb04779c9a04162957fbeb87
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/733603
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
This is the first step towards making sample clearer and less
error-prone.
- Rename Module and Package to DefaultModule and DefaultPackage, so
existing code continues to work. In later CLs, I'll replace most or
all calls to these functions.
- Define ModuleInfo, Module and Package to take as arguments the most
common fields. The Module function does _not_ create a package; it
returns a module with no packages.
- Most important: define AddPackage, which adds a Package to a Module
and correctly updates Module.Directories.
- Remove VersionedPackage, since it's clearer to create one directly.
- Update a couple of call sites, just to get tests to pass. More
updates in a later CL.
Change-Id: I46eb94ba897d4f122483b58435107b8782c6044f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/733619
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
golang.org/x/discovery is renamed to golang.org/x/pkgsite.
When the repository is open sourced, it will be hosted at
go.googlesource.com/pkgsite.
Change-Id: Ifc3b45b771a385b99179e785447f2a87afcacf87
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/724273
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Rename the type, and any variable of that type.
Rename most functions from *Version* to *Module*, where it made sense.
No other changes.
Updates b/150143758.
Change-Id: I13079411376be0af7e53a577fc6ca1c92c38b6cd
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/677160
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
This CL is pure renaming:
- internal.VersionInfo to internal.ModuleInfo
- related variables from vi to mi
Updates b/150143758.
Change-Id: I5da63a07d6f9b731016ffe92e403e5f2d9aadab7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/673875
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
The etl determines whether modules and packages are redistributable,
and saves that information to the database. The frontend just
reads the fields from the DB.
This CL replaces internal/license with internal/licenses throughout
the etl and frontend. A subsequent CL will replace it throughout
the repo, and remove the internal/license package.
Change-Id: If137d7c5e7e2ae326c28d0b5df80ddbeca134313
Fixes b/147505176.
Change-Id: I68a38baa7f55d36db9a63764571e610ca53064a6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/641370
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Previously Package.DocumentationHTML was a type []byte. This is now
changed to a type string, so that is matches its database type (which is
TEXT).
Updates b/144650806
Change-Id: I87784576e719d8aa666b0a21848a9ea6f10f8cf1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/600048
Reviewed-by: Robert Findley <rfindley@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Previously license.License.Contents was a type []byte. This is now
changed to a type string, so that is matches its database type (which is
TEXT).
Updates b/144650806
Change-Id: I4b7cb1c234ccea330d6f3ae7cf50a67ea6021693
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/600047
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Previously VersionInfo.ReadmeContents was a type []byte. This is now
changed to a type string, so that is matches its database type (which is
TEXT).
Updates b/144650806
Change-Id: I90e24dc8184b79a39f526e50ed1f75fd6d3895aa
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/600046
Reviewed-by: Robert Findley <rfindley@google.com>
Store the output of licensecheck.Cover in the licenses table.
This will make our lives easier if we ever want to revisit our
coverage parameters.
In the process, generalize scanning of jsonb columns
into Go values.
Change-Id: Iadc89c89c2590e0894fefcdcd461283d90b2e701
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/601900
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Put all test-related packages there (sample, testhelper, integration).
production risk: none
Change-Id: I06673aaa5c3fd6966410f1c1ad2b34e4f2d88f94
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599080
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Jonathan Amsterdam
Julie Qiu
Bharath Thiruveedula