Граф коммитов

153 Коммитов

Автор SHA1 Сообщение Дата
Jonathan Amsterdam 1551f083f9 internal/postgres: insert module path even if not a package
When inserting the paths of a module into the paths table, we were
inserting all the package paths and the v1 module path (the module path
without "/vN" suffix). That is almost always sufficient. But if a module
has a version suffix and is not itself a package, then we were not
inserting its path. The module path is needed by insertSymbols, so this
was causing that function to fail.

Add the module path, refactor the code to simplify it and allow testing,
and add a test.

(For a vN module the code also adds the v1 package path for every
package. I think this is unnecessary, but it's too risky to remove
it.)

Change-Id: I8caa3dd945692a4ec0a1128199237de6b77aa719
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609117
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-09-04 17:48:17 +00:00
Jonathan Amsterdam f5afe0245f several: use stdlib functions
Replace some functions with simpler code that uses
recent functions added to the standard library.

Change-Id: Ifcd570679a897fb165a4307c7ce9f0f53fccea05
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609116
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
2024-09-04 12:20:54 +00:00
Jonathan Amsterdam acdeec62f9 worker: display request state on main page
Add a RequestInfo.State field to record the current state of a request
for display.

Add a function internal.RequestState to for updating the state,
and use it for long-running requests.

Display the state on the worker main page.

Change-Id: I1a04036982aba23cf27776fac75248972a1336f3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568515
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-03-05 19:44:05 +00:00
Hana (Hyang-Ah) Kim 694b5d3c2a all: convert interface{} to any
Change-Id: I1f3b7cc8899c7707abb01e3d14807c37c3451382
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/449695
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
2022-11-15 20:10:34 +00:00
Jonathan Amsterdam 664ef41696 internal/licenses: remove Metadata.OldCoverage
We don't need the OldCoverage field anymore. All DBs have
only the new Coverage information.

Change-Id: Ib2d8e0cdacdc2f7b6514a8a5310d6905be15136c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/363954
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-11-15 20:17:11 +00:00
Jonathan Amsterdam 76f929b8d0 internal/{postgres,frontend}: support modules with zero commit times
It's valid to have a module with a zero commit time (as reported by
the proxy .info endpoint). That can happen if the module is served
from a vanity URL with a go-import meta tag of type `mod`.

Process those modules and display them with a publish date of
"unknown."

Fixes golang/go#48952

Change-Id: I8efdebfa27c8b6ed68c183bac77d87ee715f8262
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/356129
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-10-15 13:50:12 +00:00
Jonathan Amsterdam 2f714d62bb internal/postgres: remove some error logging
We were logging some information to debug a deadlock, but it was
cluttering the logs and the messages had no context viewed in
isolation. Instead just log one row.

Change-Id: I0c7c261af0371f12edf51102a0b6ec20daf4bc9b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348932
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-09-10 15:13:57 +00:00
Jonathan Amsterdam b7d799de23 internal/postgres: delete other packages from search during reconciliation
When reconciling search to match the latest good version, delete from
search_documents any packages in the module that are not in the good
version.

Change-Id: I654eda3d3066bb82963fc98ca537d4b3761ad6eb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/343189
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-18 14:32:16 +00:00
Julie Qiu 1895d95141 internal/postgres: split deleteModuleOrPackagesInModuleFromSearchDocuments
deleteModuleOrPackagesInModuleFromSearchDocuments is split into
deleteModuleFromSearchDocuments and
deletePackagesInModuleFromSearchDocuments to reduce confusion as to
which query is being executed.

Change-Id: I4559d812f54401857a153aa52166c12f228f6354
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/342631
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-08-16 21:30:56 +00:00
Jonathan Amsterdam 729fee3ca8 internal/postgres: don't insert into search if a prefix
In ReconcileSearch, don't insert a module into search_documents if
there is already a module path that is a suffix of the current one.

This avoids cases like the following:

1. github.com/araalinetworks/api, which has a golang package, is
   fetched and added to search_documents.

2. github.com/araalinetworks/api/golang is fetched and added to
   search_documents, correctly replacing github.com/araalinetworks/api
   because it is a longer module path.

3. Another version of github.com/araalinetworks/api is fetched. It is
   bad, so it isn't inserted in the usual way, but ReconcileSearch
   re-inserts the latest good version, which overwrites the longer
   module path inserted in step 2.

Change-Id: If8ae59acfde7838c0ea48882d6beb0a457e125db
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/342469
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-16 15:32:11 +00:00
Jonathan Amsterdam 46af8a773c internal/postgres: rename ReInsertLatestVersion
Rename to ReconcileSearch, which better describes its purpose.

Change-Id: Ib7562e6bf511ff1ad88129619feec110c81168d3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/342169
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-16 14:04:46 +00:00
Jonathan Amsterdam 01a72d71a6 internal/{worker,postgres}: check current module for alternative
When reconciling the contents of search_documents when we insert a
module, consider whether the version currently being processed is
alternative.

Previously, we didn't consider the current module because we only
checked the DB, and the status code for the current module hasn't been
inserted yet.

Change-Id: Ibeb02d7f56341119e2bf4f20f148b345b2b82828
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341868
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-16 14:04:40 +00:00
Jonathan Amsterdam eb20e7607d internal/{worker,postgres}: remove alternative modules from search
When processing a module, if it is an alternative module, remove it
from search_documents and imports_unique.

Change-Id: I3fd12e8522331b8a4f060c9576d97c0ee710785d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341854
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-08-13 18:10:53 +00:00
Jonathan Amsterdam ce31f46e31 internal/postgres: factor out delete from imports_unique
Add a function that deletes a module from imports_unique,
and call it where needed.

Change-Id: I0209d09855bf939e644cac646f1b8918bc48e990
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341852
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-13 14:55:28 +00:00
Jonathan Amsterdam 80022c81ae internal/postgres: remove a fully retracted module from search
If all the versions of a module have been retracted, remove it
from the search_documents table so it doesn't appear in search.

Also remove it from the imports_unique table so it doesn't contribute
to the imported-by counts of other modules.

For golang/go#47590

Change-Id: I76a6e33a1291abe4a19f1b3521c9dc8ebed4f110
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341591
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-12 19:42:01 +00:00
Jonathan Amsterdam 2b0bc991dd internal/postgres: on insert, delete only old packages from search
Instead of deleting the entire module from search_documents on insert,
just delete packages that are not in the current version. That will
avoid resetting imported-by counts to zero when we reprocess the
latest version of a module.

For golang/go#47555

Change-Id: Ie2aedc920f0e840c20f4487ed10bf7801b43a3f8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/341269
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-08-11 14:03:35 +00:00
Julie Qiu dc06dc1217 internal/postgres: move DeleteModule
DeleteModule is moved to delete.go.

Pure code in motion.

Change-Id: I2a9ab37ce4b5dd584d4163f6bf2cb2e81565c69c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/339151
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-08-02 20:54:44 +00:00
Jonathan Amsterdam bc1baae772 internal/postgres: delete old versions from search_documents on insert
When inserting the latest version of a module, delete all versions
from search_documents before inserting the latest one. That will
remove from search all packages that are not in the latest version.

Change-Id: I58fe9a1e8974d7f3fbd1fa64212c4b0381895d09
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/337751
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-07-27 16:42:59 +00:00
Jonathan Amsterdam ba35da1ee9 internal/postgres: delete alternative module from search_documents
When inserting a module whose latest version alternative, delete
all versions of the module from search_documents.

Before, we didn't insert the module into search_documents, but
we did not delete other versions of that module.

Change-Id: I87f14b883faf226be151e78605ec44cc216888b6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/334950
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-07-16 17:48:08 +00:00
Julie Qiu c1fa4f4106 internal/postgres: fix bug in isAlternativeModule
At the moment, isAlternativeModulePath does not filter on the modulePath
argument, which means that it reports whether the latest version of a
random module has status 491, and not the specified module.

This bug is fixed and a test is added.

Change-Id: I7a0feb960e1f1e6b7de2fb7a82ec0ea2fdcce738
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/332378
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-13 16:26:28 +00:00
Julie Qiu 01e3e31983 internal/postgres: change upsertSymbolSearchDocuments query
The upsertSymbolSearchDocuments query is changed to filter on module
path and version, instead of passing an array of unit_ids.

Additional tests are added to TestSymbolSearch.

For golang/go#44142

Change-Id: Ic0c493711c2c8fcc32e207ea466c509e6f99209b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329924
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-13 16:25:53 +00:00
Julie Qiu ea6abaf6b5 internal/postgres: do not insert symbols for commands
Symbols for commands don't appear in the documentation, so don't insert
them for search.

For golang/go#44142

Change-Id: I5178bfac19686245d5b5055aa38ad73c04ab9dce
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/333377
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-07-13 16:22:26 +00:00
Julie Qiu de3c212715 internal/postgres: change logic for upsertSearchDocumentSymbols
upsertSearchDocumentSymbols is changed based on the new schema.

For golang/go#44142

Change-Id: I1857b04cceeb9b702e4c54fc3a8d39efa811a3ce
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329493
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-21 17:39:06 +00:00
Julie Qiu fc9b57af4c internal/postgres: move insertSymbols to saveModule
Logic for insertSymbols is moved out of insertUnits to saveModule. This
will allows us to run that code after the check for isLatest, which is
necessary to insert symbols where the latest unit is not a
release version.

For golang/go#44142

Change-Id: Id917b6b74020957549ad1c6594f21d3f8d27e464
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329349
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-06-21 17:30:40 +00:00
Jonathan Amsterdam 115939a637 many: remove read-imports experiment
Remove the read-imports experiment and all references to the package_imports table.

Change-Id: If926c47e1d01b3ab0f31bb6efb9c2234418fca1b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329229
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-18 09:55:56 +00:00
Jonathan Amsterdam b23d3c6e62 internal/postgres: don't touch package_imports
If the read-imports experiment is on, replace all references to the
package_imports table with the imports table: don't write into or read
from package_imports.

Change-Id: I02631aba574b9938a44b06bce855997606984939
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/327749
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-14 15:29:59 +00:00
Jonathan Amsterdam d1a6ef5b25 internal/postgres: write to imports table
Write a package's import paths to the new imports table
as well as the existing package_imports table.

Change-Id: Ie0afe5aa90e79e8371a51cc6fc5f68de3dc62cd8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326449
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-10 10:49:42 +00:00
Jonathan Amsterdam 28e226e533 internal/postgres: only skip documentation_symbols
When skip-insert-symbols is on, skip only the documentation_symbols
table, not the others.

Change-Id: Ia1f6bf975f02a2cc5aad15cd220b8a6c54585e97
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/324790
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-03 17:23:30 +00:00
Jonathan Amsterdam ed260cfa67 internal/postgres: add experiment to disable symbol insertion
Inserting symbols is causing errors because the generated sequence values
overflow the integer type. Add an experiment that will let us turn off
writing to the symbols tables until we can change the type.

Change-Id: Ied3484a8db07df0f0795ff28de96760a3595ee67
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/324631
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-06-03 14:07:50 +00:00
Jonathan Amsterdam a81ea11962 internal/postgres: change symbol lock from table to path
I noticed some contention around the table lock that
upsertSymbolHistory was acquiring. (I looked at the logs of tasks that
had failed, and noticed that their last statement was the LOCK,
suggesting that they blocked on acquiring the lock and eventually
timed out.)

Instead of locking the table, we lock the module path, as we do
elsewhere for latest-version inserts. That should avoid conflicts in
upsertSymbolHistory and reduce contention, since two transactions will
only block each other if they are working on different versions of the
same module.

We acquire the lock earlier than we need to, to avoid passing
additional information down the call stack. That shouldn't matter
much.

Change-Id: I9c86e1ff6c638baade70fdee25e0ee31b30a03c4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321790
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-05-26 17:33:23 +00:00
Jonathan Amsterdam 759af54b13 internal/postgres: remove delete from documentation
Don't delete old rows from the documentation table.

Change-Id: Idd2e7d5fc02887f0fdcea4c2297b4bd499bd8ad4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/319909
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-05-13 19:53:27 +00:00
junjunjunk ab4eaf3df7 internal/postgres: use moduleID in getPackagesInUnit
Add argument moduleID to getPackagesInUnit for more efficiency.

Fixes golang/go#45854

Change-Id: Ib8c7721ae5991704c28f966c6750f17289442dc8
GitHub-Last-Rev: 4986eff558
GitHub-Pull-Request: golang/pkgsite#27
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/315489
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
2021-05-06 15:39:52 +00:00
Julie Qiu 8185298a14 internal/postgres: remove do-not-insert-new-documentation
Change-Id: I4b5e4b4e6556f6f724dd01c08b7014c30b10a2b3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/310313
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-04-21 19:14:16 +00:00
Julie Qiu cd563e35d8 internal/postgres: add exp to getDocIDsForPath
ExperimentDoNotInsertNewDocumentation is added to getDocIDsForPath, so
that we can deprecate new_documentation in a future CL.

Change-Id: I53b35d4e629974c70aec43ed0b4f124d46085dd6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/310312
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-04-16 21:12:53 +00:00
Julie Qiu 6db4e92ce5 internal/postgres: delete insert-symbols
Change-Id: I654b4b6186d49a9a8a83874747ab5ee9d909fe9a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/310380
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-04-16 19:00:33 +00:00
Julie Qiu 9f9e4bc7ca internal: add experiment do-not-insert-new-documentation
We want to rename new_documentation to documentation, and drop the
current documentation table.

INSERTs into new_documentation are added as a first step.

Change-Id: I2f4038890436946b2c140c108faa4740e16abe4e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/310375
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-04-16 02:35:07 +00:00
Jonathan Amsterdam 58432cb4a0 internal/postgres: remove uses of modules.deprecated_comment
The deprecated_column in the modules table has been superseded by the
information in the latest_module_versions table.

For golang/go#43265

Change-Id: Ib53e0b295a3edf8e807ff825b36baa6701b927b1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/309610
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-04-13 18:23:28 +00:00
Julie Qiu 5b4814e905 Revert "internal/postgres: remove documentation delete"
This reverts commit 101343b2dd.

Reason for revert: We still need to delete rows when an "all" row is
inserted to prevent panics in https://go.googlesource.com/pkgsite/+/master/internal/build_context.go#51.

Change-Id: Ic2dec11978774cde4bffeba040ef2e212385e994
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/305949
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-03-30 18:06:38 +00:00
Jonathan Amsterdam be5421da73 internal/postgres: use CopyUpsert to insert documentation
Insert into the documentation and new_documentation columns using the
new CopyUpsert method.

This saves some time. It saves even more space, because we can
generate the rows one at a time instead of making a big slice of all
the values. The difference can be quite significant on modules with
many documentation rows.

Change-Id: I859bdfaf90a204d755eb7a067da000014e0259a2
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/305831
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-30 16:42:15 +00:00
Julie Qiu 101343b2dd internal/postgres: remove documentation delete
The documentation rows for a given module no longer need to be deleted.

Instead, we are planning to recreate the documentation table by
inserting into new_documentation.

Change-Id: Ia32957fb7947e793d8f8972091d16995d70db04a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/304749
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-03-25 19:57:07 +00:00
Jonathan Amsterdam 0cbd65a06c internal/worker: call ReInsertLatestVersion
After processing a module, call ReInsertLatestVersion.

Skip the call if this is the cooked latest version of an alternative
path. The same check is made inside ReInsertLatestVersion, but using
the module_version_states table, and the module being processed has
not yet been inserted into that table.

For golang/go#44710

Change-Id: Icd0d0c0045ccadf3e97d32c63146262a3b442577
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303649
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-03-22 17:54:59 +00:00
Jonathan Amsterdam d3c0706885 internal/postgres: add imports_unique to ReInsertLatestVersion
The ReInsertLatestVersion method now changes imports_unique as well as
search_documents.

For golang/go#44710

Change-Id: Ie5dcfe8e362313be76f0ad4958882e37fc725e44
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303313
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-03-22 15:45:56 +00:00
Jonathan Amsterdam 8be65e0688 internal/postgres: add ReInsertLatestVersion
New latest-version info can retract the current latest good version,
resulting in an older version becoming the new latest good version.

Add a function that updates the search_documents table when
that happens.

A later CL will also update imports_unique, and will call this
function from the worker.

For golang/go#44710

Change-Id: I09569e27d384ed7defb06a4e81facdb96c22caeb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303312
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-22 15:40:40 +00:00
Julie Qiu be1e890ad1 internal/postgres: insert into new_documentation
Data is now inserted into the new_documentation table, which will
replace the documentation table once reprocessing is done. The
documentation_id rows for documentation_symbols now reference
new_documentation.

Tests for symbols which were deleted in CL 303116 are added back.

TestFrontendMainPage/main_page_hello_-_multi_GOOS_JS_page is flaky and
will be added back in a future CL.

Change-Id: I8d7aed2092b3d19c45c14be5f9a0986dfe224aeb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303115
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-03-19 20:20:48 +00:00
Jonathan Amsterdam 7eca3b55e9 internal/postgres: reimplement alternative-module check
When inserting a module, we skip inserting into search_documents if
the module path is an alternative one.

This CL factors out that test, and also reimplements it in terms of
the latest cooked version.  Previously, any later version with a 491
status would classify the path as alternative, but this version uses
the latest cooked (non-retracted) version's status to decide, allowing
module authors to fix mistakes.

Change-Id: I3588f3df6a1a4dd4db59333314c34167eda51df1

For golang/go#44710

Change-Id: I670d9085800fed198e6dd6469f1a8d5728c5f039
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303309
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-03-19 17:57:51 +00:00
Jonathan Amsterdam 38233971b4 internal/worker,etc.: improvements to latest-version info
- Fetch and update the latest-version info before processing a module
  rather than after, so the information in the DB matches what
  InsertModule is using.

- Pass latest-version info to InsertModule so it can use it to decide
  whether the version being inserted is the latest version.

- InsertModule writes the good latest version to the latest_module_versions
  table while holding the advisory lock, avoiding a race condition
  where two different versions both think they're the latest.

For golang/go#44710

Change-Id: Id5e8fa7a384ec6d4d86257362fd33a8fcd5215f7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/301929
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-16 18:33:03 +00:00
Julie Qiu 5f482cba8b internal/postgres: use RepeatableRead in InsertModule
The isolation level is changed back to RepeatableRead in InsertModule to
fix issues when inserting into documentation_symbols.

For golang/go#37102

Change-Id: Id8cefc8660b500f88868a0f899f90f7a97a87c7b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/298989
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-03-05 17:29:04 +00:00
Jonathan Amsterdam c35f6eebc6 internal/postgres: insert paths as a separate transaction
When inserting a module, move inserting paths to a separate
transaction that is done first.

We can then lower the isolation level of the main transaction, because
we only increased it to RepeatableRead to deal with insertPaths. See
the commit message for aada664a38.

The hope is that we'll avoid deadlocks and other problems we've seen
that all seem to be issues with the paths table.

Change-Id: I7840b87ace9237cb0f8ea6ff888f2a509034ba0b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296929
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-03-01 22:52:32 +00:00
Jonathan Amsterdam efe8624c87 internal/postgres: refactor path insertion
Move the module-independent part of insertPaths to a separate
function, upsertPaths, and put it in the paths.go file so it is near
it's single-path cousin.

Write a simple test for it.

Change-Id: I3f4a6d3e21836b2840153354c880f8be823b466b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296551
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-02-26 20:46:05 +00:00
Julie Qiu 9e8d18f876 internal/postgres: update insertDocs
insertDocs now inserts new_goos and new_goarch. It returns a map of
pathToDocIDToDoc.

For golang/go#37102

Change-Id: Ifdeead539709427c445c5a106689c1208617fa65
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/296190
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-02-25 00:10:37 +00:00