source.Client is added, which replaces the used of http.DefaultClient
when fetching source code. This also allows us to set a custom timeout
for tests vs when running the worker.
Change-Id: I5b7b0fd32fa7a2cf836b951af1ad471751fade00
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/709198
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This CL only renames directories, and makes the minimal
changes needed for tests to pass.
Updates b/150864416.
Change-Id: Iebe977a1a80341e170c306e58851fa1b48ef79e6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/686700
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
I ran unparam (`go get mvdan.cc/unparam`) on our codebase, and
it found a few issues that I fixed.
- etl/main.go:getRedis: Use the context to set the DialTimout option when dialing redis.
- config.go:readOverrideFile: Use the bucketName argument, not a global constant.
- various: Remove unused parameters and return values.
Change-Id: I98a1e5d728d41033b5f632ccc6c22442d91c2b6d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/678587
Reviewed-by: Julie Qiu <julieqiu@google.com>
Rename the type, and any variable of that type.
Rename most functions from *Version* to *Module*, where it made sense.
No other changes.
Updates b/150143758.
Change-Id: I13079411376be0af7e53a577fc6ca1c92c38b6cd
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/677160
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
This CL is pure renaming:
- internal.VersionInfo to internal.ModuleInfo
- related variables from vi to mi
Updates b/150143758.
Change-Id: I5da63a07d6f9b731016ffe92e403e5f2d9aadab7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/673875
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
To be in accordance with golang.org/wiki/CSSStyleGuide
While not noted in the CSS style guide, using the ID attribute should
only be used when a class won’t suffice (linking to a place in the page,
label “for” attribute values, etc.). Otherwise it ends up on the window
object, polluting the global namespace.
Change-Id: Ibc4954d72455b89cc284d5baa87711c9333b83fb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/672138
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
As per the bug listed below, we are trying to remove the global cfg
variable from the config package. This CL brings us close to that
goal. There are still two uses of the global that are harder to remove.
Updates b/145301722.
Change-Id: Ia6c350ef5d817ff7fb3125cd207343d10911d64f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/672580
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
The code for creating a new package is moved to internal/queue. This
allows for adding items to a task queue from the frontend, without
importing the entire internal/etl package.
No logic is changed, except the signature of
queue.InMemoryQueue.process, so that etl.FetchAndUpdateState can be
passed in, and the entire internal/etl package does not need to be
imported by internal/queue, which would lead to a circular import.
Updates b/135954292
Change-Id: I33a6418c73d85e15c4aa5593d633e19eb7e4eb1b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/671014
Reviewed-by: Jonathan Amsterdam <jba@google.com>
- Check additional packages.
- Reorganize to support checking a list of packages.
- Add a LicenseFilePath field to pagecheck.Page to correctly handle
checking the license link.
Change-Id: If5c6531ad7ad79a5093eb3c5bfca94ddbcffd398
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/640540
Reviewed-by: Julie Qiu <julieqiu@google.com>
The etl determines whether modules and packages are redistributable,
and saves that information to the database. The frontend just
reads the fields from the DB.
This CL replaces internal/license with internal/licenses throughout
the etl and frontend. A subsequent CL will replace it throughout
the repo, and remove the internal/license package.
Change-Id: If137d7c5e7e2ae326c28d0b5df80ddbeca134313
Fixes b/147505176.
Change-Id: I68a38baa7f55d36db9a63764571e610ca53064a6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/641370
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
fetch.Fetch now returns the module path from the go.mod file, and
the ETL writes it to the DB.
Along the way, simplify the return values of Fetch with a struct.
Change-Id: I191129b37c397ecb00e17aa7ba39e9cb0d0c89b6
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/632524
Reviewed-by: Julie Qiu <julieqiu@google.com>
Read the go.mod file from the proxy, and check if its path differs
from the module path. If they do differ, fail with an
AlternativeModule error, and record the relationship between the two
paths in a table for future use.
This change means that the discovery site will not serve modules that:
- are case variants of the canonical path
e.g. github.com/Sirupsen/logrus vs. the correct github.com/sirupsen/logrus
- bypass the vanity import path
e.g. github.com/gonum/gonum vs. the correct gonum.org/v1/gonum
- are "soft forks" of the module's repo: forks with an unchanged go.mod path
e.g. github.com/alice02/kubernetes vs. the correct k8s.io/kubernetes
Change-Id: If78c94744440112f5720750885fe10f6a7dc7ab8
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/623922
Reviewed-by: Julie Qiu <julieqiu@google.com>
We currently import internal/etl into internal/proxydatasource,
because etl.FetchVersion is used by the proxydatasource.
These functions are moved into their own package fetch.
FetchAndInsertVersion may be moved back to internal/etl in a later CL,
since it is only used by the ETL. At the moment, all functionality for
FetchVersion is being tested by FetchAndInsertVersion.
Updates b/146890756
Change-Id: Icaaf1feeac3821c82a85f21e07085d65556328d5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/626675
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
The same can be achieved with CSS selectors, using :nth-child
and :nth-of-type.
Change-Id: I57486922d5d0060862b20d67f56be808b6c65082
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/617945
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Previously, we were relying on GetPackage returning an error in order
for directories not to show the latest badge. Aside from making an
unnecessary database call, there are some cases where we're serving a
directory for a path, but that path is a valid package at another
version, so GetPackage(..., "latest") actually succeeds. In fact, it's
possible that the latest valid package version is actually *earlier*
than the directory version we're currently serving, which is especially
confusing for the user.
In order to be consistent about showing the badge only on module and
package pages, a new field 'PageType' is introduced on Details to expose
exactly which details handler is serving the page. This is then used in
the latestVersion function.
The new PageType field holds strictly more information than the existing
Namespace field, so Namespace was removed and all uses were updated to
use PageType.
Additionally, the order of 'packagePath' and 'modulePath' in the
latestVersion function was reversed. In all other argument lists
packagePath precedes modulePath (typically pkgPath, modulePath,
version), so this was changed purely for consistency.
Fixes b/143814014
Change-Id: I3d9fd40c44f0d3f02c38755950c3988699a00891
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/616656
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
servePackagePage is updated to prefer serving a directory at the
requested version if it exists. This fixes the bug where the breadcrumb
link to github.com/docker/docker fails at any version > 0.9.1 (the
latest version where github.com/docker/docker was a package).
Broken tests in TestModulePackageDirectoryResolution are uncommented.
Notably, the test for github.com/golang/found/dir@latest is updated to
reflect revised behavior: in this case we will serve the latest package.
More discussion of this in
Updates b/143814014
Change-Id: I2822fef16259eb842aeba0c2f60c975a599b3999
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/616023
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
A new integration test, TestModulePackageDirectoryResolution is added to
test the resolution of modules vs. packages vs. directories when a path
is only a valid package a certain versions.
In order for this test to pass I had to fix the return code when we
suggest a search: previously we were returning StatusSeeOther, but that
should really only be used when we're actually redirecting to a
different page via the location header. Instead, we now return
StatusNotFound.
A convenience function htmlcheck.Run is added for running htmlcheck on
the Body of a net/http.Response.
Updates b/143814014
Fixes b/145858138
Updates b/144031201
Updates b/143760329
Change-Id: Ie3f02165a6607b341379a1aea2eb078f300ad405
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/614811
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
null was being returned when completion results are empty, which causes
autocomplete.js to generate errors.
In order to test this, hook up an HA redis to the integration test and
populate auto-completions.
Updates b/143370178
Change-Id: Ice30c7a6f9f3d4907746103193947bbd5dc5a45b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/607126
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Previously Package.DocumentationHTML was a type []byte. This is now
changed to a type string, so that is matches its database type (which is
TEXT).
Updates b/144650806
Change-Id: I87784576e719d8aa666b0a21848a9ea6f10f8cf1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/600048
Reviewed-by: Robert Findley <rfindley@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Previously license.License.Contents was a type []byte. This is now
changed to a type string, so that is matches its database type (which is
TEXT).
Updates b/144650806
Change-Id: I4b7cb1c234ccea330d6f3ae7cf50a67ea6021693
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/600047
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Previously VersionInfo.ReadmeContents was a type []byte. This is now
changed to a type string, so that is matches its database type (which is
TEXT).
Updates b/144650806
Change-Id: I90e24dc8184b79a39f526e50ed1f75fd6d3895aa
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/600046
Reviewed-by: Robert Findley <rfindley@google.com>
Put the link in the latest-version badge in the form
mod@version/suffix.
The best way to do this was to move the latest-badge-link
logic from the template to the server itself.
Also, rename the Suffix field of Package to PathAfterDirectory
to more accurately describe it.
Fixes b/144681389.
Change-Id: I16a78224ee6fa53b60954716338347a298dbef21
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/604414
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Extract into a separate package the core functionality from
internal/postgres that doesn't depend on our particular schema.
This makes it available for other uses, like devtools commands and etl
autocomplete.
Do the same for testing functionality.
We now have three packages where before we had only one:
- internal/postgres: discovery-specific DB operations and test support
- internal/database: discovery-agnostic DB operations
- internal/testing/dbtest: discovery-agnostic DB test support
Change-Id: I54c59aee328dae71ba6c77170a72e7a83da7c785
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602327
Reviewed-by: Robert Findley <rfindley@google.com>
Now that completion data is being populated by the ETL, we can use it to
provide a JSON API for search autocompletions at /autocomplete?q=query.
This API reads lexical completions from Redis, sorts according to a
scoring heuristic, and returns the results as a JSON array.
For testing purposes, the pathCompletions function is moved from the etl
package to the complete package, and exported.
Updates b/143370178
Change-Id: I61d75296399222fa9ebfe8aadc909bf048389a23
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602064
Reviewed-by: Jonathan Amsterdam <jba@google.com>
We're going to compute auto-completions using redis sorted sets in the
'HA' redis instance (because we don't want them to be evicted by
caching). This data need not be very fresh, and so can be computed by
the ETL every so often, perhaps once a day.
This CL wires the HA redis instance into the ETL, and adds an action to
update these Redis sorted sets. Doing this takes 1-2 minutes and
consumes quite a lot of memory, so there are some optimizations used:
+ the underlying *sql.DB is used for streaming search documents, rather
than a method on postgres.DB. There is explanation for this in the
code, and additionally I don't think this is so bad as long as it is
only done in the ETL.
+ Completions data is loaded into temporary Redis keys and then renamed
to overwrite the keys that will be used by the frontend, in order to
minimize disruption while the completion data is being updated.
Updates b/143370178
Change-Id: I48caf77eb26d8ea7b60c2e117481bfa98ef089ca
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602062
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Avoid the use of the naked term "version" in the details
header. Instead, we have:
- DisplayVersion: the version formatted for human readability.
- LinkVersion: the version to use in links to the discovery site.
Also, use the link version in nav tab links:
Fixes b/144873325.
And improve test coverage.
Change-Id: Ida80867b6bd70a32c18f45b21b0b361b9e8e904f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/603789
Reviewed-by: Robert Findley <rfindley@google.com>
Store the output of licensecheck.Cover in the licenses table.
This will make our lives easier if we ever want to revisit our
coverage parameters.
In the process, generalize scanning of jsonb columns
into Go values.
Change-Id: Iadc89c89c2590e0894fefcdcd461283d90b2e701
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/601900
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
A recent change (not yet in prod) introduced a bug:
On the overview page of a module with a pseudoversion, the
module link has the formatted version string (e.g. "v0.0.0 (201909091-392039)")
instead of the actual version.
Fixed, and added server tests to check.
Change-Id: I9f88272e45f496f21d741a3f119396c7223c7ee9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/601903
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add data-test-id attributes to the license and module parts
of the info label, so we can more reliably extract them
for tests.
This practice is suggested by the TotT "Select DOM Elements by User-Facing Properties"
Change-Id: Ie4bf1cac8c3e88b728c2561256d210ab58819968
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/600820
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Factor out common structural checks on pages, so we can
use them in more than one place.
- Add internal/testing/pagecheck for common checks on page contents
- Use in internal/frontend/server_test.go
Change-Id: I744cf090f6bfad8c788be5b21cb5ed80f9adf8de
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/600800
Reviewed-by: Julie Qiu <julieqiu@google.com>
Move the html-checking code from internal/frontend tests to its own package.
Ths is a first step towards making these structural tests usable elsewhere,
like in an integration test that runs against a live endpoint.
Since these are now exported, give them better names in some cases.
Also, generalize to use regexps where that makes sense.
Change-Id: I1a3e0c09954d9cb07200c087db17e499a9d0d9a7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599089
Reviewed-by: Julie Qiu <julieqiu@google.com>
Put all test-related packages there (sample, testhelper, integration).
production risk: none
Change-Id: I06673aaa5c3fd6966410f1c1ad2b34e4f2d88f94
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599080
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Julie Qiu
Andrew Bonventre
Jonathan Amsterdam
Rob Findley