Instrumentation for frontend fetch requests is added via the ochttp and
ocsql integrations.
Change-Id: I095ec224c24bdb7e341f2b620ddf020ed50f556e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/765147
Reviewed-by: Jonathan Amsterdam <jba@google.com>
We use zip files as license test data. They should not be ignored by
git.
Change-Id: I21672c4fae5b759c8a9869b63f465c8550d8b513
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/769679
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Also, fix bug where the worker change interval was being passed in.
Change-Id: Ia1fa19b231acd2d0d307ea418d1fd0d8c3a4c320
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/768541
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Instead of returning an error if a package's documentation is too
large, replace the documentation with a short string that says it's
too big, and proceed.
Fixes b/158556376.
Change-Id: Ib9754da73eea4b60bcbe422e122b2188b6ae668a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766359
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Waiting for subsequent searchers to complete after the first one fails
behaves poorly in overload scenarios.
For example, one of the ways that overload can manifest itself is by
connection exhaustion or Postgres actively refusing connections. Let's
assume this is happening, and Postgres randomly refuses half the
connections.
Now let's assume that we have a particular query for a rare term that is
much faster and cheaper when using the reverse index. To be concrete,
let's say that 'deep' search (using the index) costs 0.1, and 'popular'
search (doing a linear scan) costs 10 -- the units don't matter.
Normally, the expected query cost of this search is approximately 0.2:
0.1 for the deep search, and around 0.1 for the popular search that was
simultaneously occurring until deep completed.
If queries are randomly refused with a 50% probability, 25% the time
this query is going to cost 0.1, because popular search was refused and
deep succeeds, 25% of the time it's going to cost 10 (!), because deep
search was refused and we continued to wait for popular search to
complete, 25% of the time it costs ~0 because both searches fail, and
25% of the time it costs 0.2 as usual, because they both succeed.
That means that in this particular scenario, our query went from an
expected cost of 0.2, to an expected cost of ~2.6, an order of magnitude
more! Not a good thing to happen while the database is already
overloaded.
Instead, let's error out the search on the first error from any
searcher.
A test is added to exercise the behavior of hedgedSearch in the presence
of errors.
Also fix some miscellaneous stale comments along the way.
Change-Id: Id0496b3d402a403a17160c63edaacad537302284
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766809
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
At the moment there are several methods not implemented by the
proxydatasource. These are removed from internal.DataSource.
Fixes b/150138536
Change-Id: Id1eef4b2497bd46c8e3cecf8a083ef81bfbe2f47
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/768688
Reviewed-by: Jonathan Amsterdam <jba@google.com>
It was hard to read the args to NewServer, and now that we have two
redis clients, it was error-prone to provide them (order matters).
Using a struct effectively lets the caller name the args.
Change-Id: I0e2e39e09402031fd21a754961a2685c377c75fc
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/768540
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
A diagram and additional details are added about the architecture of
pkg.go.dev.
Change-Id: I87cf5ed86fa9dcce5a9aa7656b6731a672fdd7eb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/768311
Reviewed-by: Jamal Carvalho <jamalcarvalho@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Fix staticcheck error:
internal/worker/server.go:491:10: error strings should not be capitalized (ST1005)
Change-Id: I7307e897fc7dc6e77075aca96f1fd15a5a338a33
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/768314
Reviewed-by: Jonathan Amsterdam <jba@google.com>
We sometimes want to clear our Redis-based cache. Adding an endpoint
to the worker will let us do this more simply and with less chance of
error than running the redis CLI.
Change-Id: I855ea5d906f0cb080b4e2f6d5fa279a6a2e0b949
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/768539
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
ModuleInfo is added, which represents the module info for the new data
model. ModuleInfo is embedded in LegacyModuleInfo.
In order to support the existing overview tab functionality,
GetDirectoryNew was changed to return the module README, regardless of
whether there is a README for the directory.
We will change this logic to display the README for the directory in a
future CL.
Change-Id: I624a6d99b711870826fd7dff9100d4ad47852db2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766801
Reviewed-by: Jonathan Amsterdam <jba@google.com>
README.md, CONTRIBUTING.md, doc/design.md and doc/frontend.md are
updated improve clarity on the project direction and how to contribute
to the project.
Change-Id: Ia4b7302cf4851c0f777f8b7c3fa45c60dad7d989
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766270
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When we insert module information, make sure we overwrite existing
rows. Previously, we specified that new data should be ignored if the
row already existed, which meant that we were adding rows but never
changing them. That didn't matter at one point, when we deleted a
module before re-inserting it, but we no longer delete, so we must
upsert.
We also fix a number of places where our test modules had duplicate
directories. The upserts failed for these.
Change-Id: I97465b11e4ea6cbb7835e883f36f098445207eba
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766365
Reviewed-by: Julie Qiu <julieqiu@google.com>
Parse the env var value carefully. In particular, an empty env var means
"accept nothing", not "accept the empty string".
Fixes b/157403867.
Change-Id: I67d98da0c756af7517492e3d904d22fe64136113
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766370
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Add DB.BulkUpsert, which adds an ON CONFLICT clause to the INSERT
that replaces existing column values.
Change-Id: I59f36be0bcb0c0854f42da489e265f2a1396c439
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766360
Reviewed-by: Julie Qiu <julieqiu@google.com>
ModuleInfo is renamed to LegacyModuleInfo, as a step towards
deprecating LegacyReadmeFilePath and LegacyReadmeContents.
In a follow up CL, we will add ModuleInfo as an embedded struct
to LegacyModuleInfo.
Change-Id: Ie452420448eec1d13edaf62e548df0b9e2cbbe4b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766479
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This change introduces a new method on Renderer, ShortSynopsis. It
is meant to be used in templates when rendering methods and functions
within the side navigation UI component.
Due to the limited horizontal space available, it omits the “func”
keyword, type information, and return values.
An example of functions and methods from the time package rendered
using this new method is below.
Functions
After(d)
Sleep(d)
Tick(d)
Types
type Duration
ParseDuration(s)
Since(t)
Until(t)
(d) Hours()
(d) Microseconds()
(d) Milliseconds()
(d) Minutes()
(d) Nanoseconds()
(d) Round(m)
(d) Seconds()
(d) String()
(d) Truncate(m)
type Location
FixedZone(name, offset)
LoadLocation(name)
LoadLocationFromTZData(name, data)
(l) String()
type Month
(m) String()
type ParseError
(e) Error()
Updates b/148095016
Change-Id: I663eaafdc0baa3619e449ccd0d8ee8d601974392
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/763562
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
We were preparing the full statement (of size stride) even if we never used
it, because the number of values to insert was less than stride.
Also, tweak TestLargeBulkInsert:
- the logging is no longer voluminous, so we don't have to turn it off;
- make the number of values not divisible by 1000, so we test the case
where rightBound > len(values).
Change-Id: I169130ab370b192a629025886173b1e33f5d158b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766240
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Some BulkInserts were being called inside a loop when they should
be outside.
Change-Id: I5c00c45c24784a69e5b273c54eecbc63a8ae0634
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766239
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
When a fetch request 404s, display the requested version in the
error message if it is not the latest version.
Change-Id: I400d753ad84290187a800e6aeb59719c572065b2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/765167
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
searchRequestRedirectPath now reads from the paths table when
isActiveUseDirectories is true.
Change-Id: I1094bac135524296f4c7b18524fd337df3581c9e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/762856
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The following structs have been renamed with a Legacy prefix:
* internal.Directory
* internal.Package
* internal.VersionedPackage
* sample.Package
The following fields on internal.ModuleInfo have also been changed:
* ReadmeFilePath
* ReadmeFileContents
This is done to help us distinguish between legacy and method
structs/methods while migrating code to the new data model.
Change-Id: Ibedf71d4db6323ef5aa05d73a0240537ea6073d3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/765160
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Previously there were two methods for serving a directory page:
* serveDirectoryPage: fetched a directory from the DB, then called
serveDirectoryPageWithDirectory
* serveDirectoryPageWithDirectory: constructs a directory page
serveDirectoryPage now implements the logic previously done by
serveDirectoryPageWithDirectory. serveDirectoryPageWithDirectory is
removed.
Change-Id: Ia749b8234f44f27b27b7c633515d5e29faa04717
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/762858
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Methods on internal.Datasource are rearranged, to group together the
legacy methods that will be deprecated. This is done to help keep track
of which methods need to be removed once the frontend migration is
complete.
Change-Id: I176743d89b8eef8f2334301f6eefafe469c30ab4
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/763560
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The experiment logic is refactored for the following:
* Frontend fetch: only requires frontend-fetch and insert-directories
experiments to be active. We can have this feature on without
everything migrating to using the paths table.
* Requests to path@master: only requires frontend-fetch,
insert-directories, and support-path-at-master.
* Use directories: requires both insert-directories and use-directories
to be active.
Change-Id: I0b24dfbcbdb4938711851554dfd13e2d2c88c645
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/762855
Reviewed-by: Jonathan Amsterdam <jba@google.com>
We were assigning an error to a dead variable (the `err` defined in the
`QueryContext` call shadowed the one above the `if`).
Caught by staticcheck.
Change-Id: Ifbf8a78c336ab1918fdf5714548b286b73199ce9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/765142
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Log how much memory we're using at various points while inserting a
module. We want to know where we consume a lot of memory, to prevent
OOMing. See bug for context.
Updates b/158067614.
Change-Id: Icf730cad039737037a0766d2ad0cfc809b7ec3e5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/763039
Reviewed-by: Julie Qiu <julieqiu@google.com>
At the moment when a request for path@version 404s, but other versions
of the path exists, we will return an error message letting users know
that is the case. Once frontend fetch is live, the message will change
to provide users an option to fetch that version of the path.
This logic is moved to pathFoundAtLatestError.
Updates golang/go#36811
Updates golang/go#37002
Updates golang/go#37106
Change-Id: I1ad15ee13714a68b4b88fd353e43719fda0c0d31
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/754818
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When a path page 404s, it will now render the notfound.tmpl page (once
the frontend-fetch feature flag is on), which provides a button for the
user to make a request to fetch the package.
Updates golang/go#36811
Updates golang/go#37002
Updates golang/go#37106
Change-Id: I17fedd018435e8d3e51e2a2a4a972d3cf673df56
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/753606
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
isSupportedVersion is added. master is now a supported version if the
feature flag ExperimentFrontendPackageAtMaster is on.
Updates golang/go#36811
Change-Id: Ib1c885a5e75cb697b8e99fd8e8c9a347e2b344ed
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/753605
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
A fetch endpoint is added to the frontend, which will queue a module to
be enqueued by the worker, if it doesn't already exist in the database.
After enqueuing, the fetch handler will poll the version_map table at a
constant rate, until the path returns or the request times out.
If the request fails, a corresponding statusCode and responseText will
be returned to be displayed to the user.
Updates golang/go#36811
Updates golang/go#37002
Updates golang/go#37106
Change-Id: Ic2e20146dc626bf296db05bc2abbfb50d6fd7991
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/743103
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Instructions are added for how to install golang-migrate. Additional context is added for how to populate the local database, and run the frontend and worker servers.
Updates b/157994345
Change-Id: I937fd77da5894c01f2c380cc94055fa786a448ce
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/760281
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Using a prepared statement for the query seems to speed up an insert significantly.
Change-Id: I32a1455b376e08fb435b2e756aa8df4b0c2bc1b3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/761661
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Change UpdateExperiment to return NotFound if the experiment name
isn't in the DB.
Change-Id: I7db11e2d2ba5a13824280e88d1ac6c22c860a7e7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/763043
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Instead of serializable transactions, we will use an advisory
lock on the module path. See http://tg/760100.
Change-Id: Iaae7777bab8efa21347480b6f3929a1feed98e44
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/762240
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Julie Qiu
Jonathan Amsterdam
Rob Findley
Andrew Bonventre
Miguel Acero