When inserting the paths of a module into the paths table, we were
inserting all the package paths and the v1 module path (the module path
without "/vN" suffix). That is almost always sufficient. But if a module
has a version suffix and is not itself a package, then we were not
inserting its path. The module path is needed by insertSymbols, so this
was causing that function to fail.
Add the module path, refactor the code to simplify it and allow testing,
and add a test.
(For a vN module the code also adds the v1 package path for every
package. I think this is unnecessary, but it's too risky to remove
it.)
Change-Id: I8caa3dd945692a4ec0a1128199237de6b77aa719
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609117
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Replace some functions with simpler code that uses
recent functions added to the standard library.
Change-Id: Ifcd570679a897fb165a4307c7ce9f0f53fccea05
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609116
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Add a RequestInfo.State field to record the current state of a request
for display.
Add a function internal.RequestState to for updating the state,
and use it for long-running requests.
Display the state on the worker main page.
Change-Id: I1a04036982aba23cf27776fac75248972a1336f3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568515
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
We don't need the OldCoverage field anymore. All DBs have
only the new Coverage information.
Change-Id: Ib2d8e0cdacdc2f7b6514a8a5310d6905be15136c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/363954
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
It's valid to have a module with a zero commit time (as reported by
the proxy .info endpoint). That can happen if the module is served
from a vanity URL with a go-import meta tag of type `mod`.
Process those modules and display them with a publish date of
"unknown."
Fixesgolang/go#48952
Change-Id: I8efdebfa27c8b6ed68c183bac77d87ee715f8262
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/356129
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
We were logging some information to debug a deadlock, but it was
cluttering the logs and the messages had no context viewed in
isolation. Instead just log one row.
Change-Id: I0c7c261af0371f12edf51102a0b6ec20daf4bc9b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348932
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
When reconciling search to match the latest good version, delete from
search_documents any packages in the module that are not in the good
version.
Change-Id: I654eda3d3066bb82963fc98ca537d4b3761ad6eb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343189
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
deleteModuleOrPackagesInModuleFromSearchDocuments is split into
deleteModuleFromSearchDocuments and
deletePackagesInModuleFromSearchDocuments to reduce confusion as to
which query is being executed.
Change-Id: I4559d812f54401857a153aa52166c12f228f6354
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/342631
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
In ReconcileSearch, don't insert a module into search_documents if
there is already a module path that is a suffix of the current one.
This avoids cases like the following:
1. github.com/araalinetworks/api, which has a golang package, is
fetched and added to search_documents.
2. github.com/araalinetworks/api/golang is fetched and added to
search_documents, correctly replacing github.com/araalinetworks/api
because it is a longer module path.
3. Another version of github.com/araalinetworks/api is fetched. It is
bad, so it isn't inserted in the usual way, but ReconcileSearch
re-inserts the latest good version, which overwrites the longer
module path inserted in step 2.
Change-Id: If8ae59acfde7838c0ea48882d6beb0a457e125db
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/342469
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
When reconciling the contents of search_documents when we insert a
module, consider whether the version currently being processed is
alternative.
Previously, we didn't consider the current module because we only
checked the DB, and the status code for the current module hasn't been
inserted yet.
Change-Id: Ibeb02d7f56341119e2bf4f20f148b345b2b82828
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341868
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
When processing a module, if it is an alternative module, remove it
from search_documents and imports_unique.
Change-Id: I3fd12e8522331b8a4f060c9576d97c0ee710785d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341854
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Add a function that deletes a module from imports_unique,
and call it where needed.
Change-Id: I0209d09855bf939e644cac646f1b8918bc48e990
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341852
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
If all the versions of a module have been retracted, remove it
from the search_documents table so it doesn't appear in search.
Also remove it from the imports_unique table so it doesn't contribute
to the imported-by counts of other modules.
For golang/go#47590
Change-Id: I76a6e33a1291abe4a19f1b3521c9dc8ebed4f110
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341591
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Instead of deleting the entire module from search_documents on insert,
just delete packages that are not in the current version. That will
avoid resetting imported-by counts to zero when we reprocess the
latest version of a module.
For golang/go#47555
Change-Id: Ie2aedc920f0e840c20f4487ed10bf7801b43a3f8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341269
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
When inserting the latest version of a module, delete all versions
from search_documents before inserting the latest one. That will
remove from search all packages that are not in the latest version.
Change-Id: I58fe9a1e8974d7f3fbd1fa64212c4b0381895d09
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/337751
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
When inserting a module whose latest version alternative, delete
all versions of the module from search_documents.
Before, we didn't insert the module into search_documents, but
we did not delete other versions of that module.
Change-Id: I87f14b883faf226be151e78605ec44cc216888b6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/334950
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
At the moment, isAlternativeModulePath does not filter on the modulePath
argument, which means that it reports whether the latest version of a
random module has status 491, and not the specified module.
This bug is fixed and a test is added.
Change-Id: I7a0feb960e1f1e6b7de2fb7a82ec0ea2fdcce738
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/332378
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The upsertSymbolSearchDocuments query is changed to filter on module
path and version, instead of passing an array of unit_ids.
Additional tests are added to TestSymbolSearch.
For golang/go#44142
Change-Id: Ic0c493711c2c8fcc32e207ea466c509e6f99209b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329924
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Symbols for commands don't appear in the documentation, so don't insert
them for search.
For golang/go#44142
Change-Id: I5178bfac19686245d5b5055aa38ad73c04ab9dce
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333377
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
upsertSearchDocumentSymbols is changed based on the new schema.
For golang/go#44142
Change-Id: I1857b04cceeb9b702e4c54fc3a8d39efa811a3ce
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329493
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Logic for insertSymbols is moved out of insertUnits to saveModule. This
will allows us to run that code after the check for isLatest, which is
necessary to insert symbols where the latest unit is not a
release version.
For golang/go#44142
Change-Id: Id917b6b74020957549ad1c6594f21d3f8d27e464
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329349
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Remove the read-imports experiment and all references to the package_imports table.
Change-Id: If926c47e1d01b3ab0f31bb6efb9c2234418fca1b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329229
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
If the read-imports experiment is on, replace all references to the
package_imports table with the imports table: don't write into or read
from package_imports.
Change-Id: I02631aba574b9938a44b06bce855997606984939
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/327749
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Write a package's import paths to the new imports table
as well as the existing package_imports table.
Change-Id: Ie0afe5aa90e79e8371a51cc6fc5f68de3dc62cd8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326449
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
When skip-insert-symbols is on, skip only the documentation_symbols
table, not the others.
Change-Id: Ia1f6bf975f02a2cc5aad15cd220b8a6c54585e97
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/324790
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Inserting symbols is causing errors because the generated sequence values
overflow the integer type. Add an experiment that will let us turn off
writing to the symbols tables until we can change the type.
Change-Id: Ied3484a8db07df0f0795ff28de96760a3595ee67
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/324631
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
I noticed some contention around the table lock that
upsertSymbolHistory was acquiring. (I looked at the logs of tasks that
had failed, and noticed that their last statement was the LOCK,
suggesting that they blocked on acquiring the lock and eventually
timed out.)
Instead of locking the table, we lock the module path, as we do
elsewhere for latest-version inserts. That should avoid conflicts in
upsertSymbolHistory and reduce contention, since two transactions will
only block each other if they are working on different versions of the
same module.
We acquire the lock earlier than we need to, to avoid passing
additional information down the call stack. That shouldn't matter
much.
Change-Id: I9c86e1ff6c638baade70fdee25e0ee31b30a03c4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321790
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Don't delete old rows from the documentation table.
Change-Id: Idd2e7d5fc02887f0fdcea4c2297b4bd499bd8ad4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/319909
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
ExperimentDoNotInsertNewDocumentation is added to getDocIDsForPath, so
that we can deprecate new_documentation in a future CL.
Change-Id: I53b35d4e629974c70aec43ed0b4f124d46085dd6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/310312
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
We want to rename new_documentation to documentation, and drop the
current documentation table.
INSERTs into new_documentation are added as a first step.
Change-Id: I2f4038890436946b2c140c108faa4740e16abe4e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/310375
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The deprecated_column in the modules table has been superseded by the
information in the latest_module_versions table.
For golang/go#43265
Change-Id: Ib53e0b295a3edf8e807ff825b36baa6701b927b1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/309610
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Insert into the documentation and new_documentation columns using the
new CopyUpsert method.
This saves some time. It saves even more space, because we can
generate the rows one at a time instead of making a big slice of all
the values. The difference can be quite significant on modules with
many documentation rows.
Change-Id: I859bdfaf90a204d755eb7a067da000014e0259a2
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/305831
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
The documentation rows for a given module no longer need to be deleted.
Instead, we are planning to recreate the documentation table by
inserting into new_documentation.
Change-Id: Ia32957fb7947e793d8f8972091d16995d70db04a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/304749
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
After processing a module, call ReInsertLatestVersion.
Skip the call if this is the cooked latest version of an alternative
path. The same check is made inside ReInsertLatestVersion, but using
the module_version_states table, and the module being processed has
not yet been inserted into that table.
For golang/go#44710
Change-Id: Icd0d0c0045ccadf3e97d32c63146262a3b442577
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303649
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
The ReInsertLatestVersion method now changes imports_unique as well as
search_documents.
For golang/go#44710
Change-Id: Ie5dcfe8e362313be76f0ad4958882e37fc725e44
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303313
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
New latest-version info can retract the current latest good version,
resulting in an older version becoming the new latest good version.
Add a function that updates the search_documents table when
that happens.
A later CL will also update imports_unique, and will call this
function from the worker.
For golang/go#44710
Change-Id: I09569e27d384ed7defb06a4e81facdb96c22caeb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303312
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
Data is now inserted into the new_documentation table, which will
replace the documentation table once reprocessing is done. The
documentation_id rows for documentation_symbols now reference
new_documentation.
Tests for symbols which were deleted in CL 303116 are added back.
TestFrontendMainPage/main_page_hello_-_multi_GOOS_JS_page is flaky and
will be added back in a future CL.
Change-Id: I8d7aed2092b3d19c45c14be5f9a0986dfe224aeb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303115
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When inserting a module, we skip inserting into search_documents if
the module path is an alternative one.
This CL factors out that test, and also reimplements it in terms of
the latest cooked version. Previously, any later version with a 491
status would classify the path as alternative, but this version uses
the latest cooked (non-retracted) version's status to decide, allowing
module authors to fix mistakes.
Change-Id: I3588f3df6a1a4dd4db59333314c34167eda51df1
For golang/go#44710
Change-Id: I670d9085800fed198e6dd6469f1a8d5728c5f039
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303309
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
- Fetch and update the latest-version info before processing a module
rather than after, so the information in the DB matches what
InsertModule is using.
- Pass latest-version info to InsertModule so it can use it to decide
whether the version being inserted is the latest version.
- InsertModule writes the good latest version to the latest_module_versions
table while holding the advisory lock, avoiding a race condition
where two different versions both think they're the latest.
For golang/go#44710
Change-Id: Id5e8fa7a384ec6d4d86257362fd33a8fcd5215f7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/301929
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
The isolation level is changed back to RepeatableRead in InsertModule to
fix issues when inserting into documentation_symbols.
For golang/go#37102
Change-Id: Id8cefc8660b500f88868a0f899f90f7a97a87c7b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298989
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
When inserting a module, move inserting paths to a separate
transaction that is done first.
We can then lower the isolation level of the main transaction, because
we only increased it to RepeatableRead to deal with insertPaths. See
the commit message for aada664a38.
The hope is that we'll avoid deadlocks and other problems we've seen
that all seem to be issues with the paths table.
Change-Id: I7840b87ace9237cb0f8ea6ff888f2a509034ba0b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296929
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Move the module-independent part of insertPaths to a separate
function, upsertPaths, and put it in the paths.go file so it is near
it's single-path cousin.
Write a simple test for it.
Change-Id: I3f4a6d3e21836b2840153354c880f8be823b466b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296551
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
insertDocs now inserts new_goos and new_goarch. It returns a map of
pathToDocIDToDoc.
For golang/go#37102
Change-Id: Ifdeead539709427c445c5a106689c1208617fa65
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296190
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>