Граф коммитов

3253 Коммитов

Автор SHA1 Сообщение Дата
Weida Hong 7d0ea0a0e8 internal/godoc/dochtml: enhance output parsing for example func
Current logic for detecting "concluding line comment" is not strict
enough that it may cause false-positive, causing some normal comment
(and code following it) to be discarded in the rendered web page.

Ensure that "concluding line comment" can only be prefixed by spaces.
Also add corresponding test case.

Fixes golang/go#65450

Change-Id: Ief9ec5326aa94965ca02d3278398cc5663ba395f
GitHub-Last-Rev: 7312ff3dff
GitHub-Pull-Request: golang/pkgsite#93
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/600415
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
2024-09-13 13:08:37 +00:00
Jonathan Amsterdam 47024e5792 internal/godoc/dochtml/internal/render: unique headings redux
This CL takes an alternative approach to generate unique headings.
It builds on the work of https://go.dev/cl/573595,
which was a fix for https://go.dev/issue/64582.

- It takes a more direct approach to constructing a unique string
  from an ast.Decl.

- The function that does that is tested separately, reducing
  the test cases needed for formatDocHTML.

- It saves the generated headings, so the HTML doesn't have to
  be reparsed.

NOTE: This will break all links to headings that are not in the package
comment. Happily, such headings are rare: only 40 of the top thousand
packages have one. It might seem we could avoid any breakage by only
applying a suffix to duplicate headings. But then at any time
in the future, a unique heading could become a duplicate, causing
a break. Better one break now than an unending stream of them.

Change-Id: I379712b54c6bc9c6a9343d0006639085a40e23d9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/608035
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2024-09-10 17:35:46 +00:00
Jonathan Amsterdam e0bd492bb7 internal/fetch,godoc: handle lint exceptions
This CL addresses the errors found by staticcheck when we upgraded
to Go 1.23. All were deprecation warnings, and they were silenced
by "ignore" directives until we could deal with them.

In all cases, ignore the lint message was the right choice.
The CL adds justifications.

Change-Id: I09d4bb96cb9913f8e35189ed3a4af9deaa092447
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/610801
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-09-05 13:10:54 +00:00
Tim King 6b577b411e all: fix printf(var) mistake detected by latest printf checker
For golang/go#69267.

Change-Id: I050c167e2e3ce36541e0d71469fb3f5778f5a882
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/610517
Reviewed-by: Ian Lance Taylor <iant@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-09-05 03:04:40 +00:00
Jonathan Amsterdam 0709f9d1cc internal/godoc/codec: improve documentation
Change-Id: I7b2d5721ee5502897be4d49d8c69ed16ee33fa8d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/610715
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-09-04 23:28:05 +00:00
Jonathan Amsterdam 1551f083f9 internal/postgres: insert module path even if not a package
When inserting the paths of a module into the paths table, we were
inserting all the package paths and the v1 module path (the module path
without "/vN" suffix). That is almost always sufficient. But if a module
has a version suffix and is not itself a package, then we were not
inserting its path. The module path is needed by insertSymbols, so this
was causing that function to fail.

Add the module path, refactor the code to simplify it and allow testing,
and add a test.

(For a vN module the code also adds the v1 package path for every
package. I think this is unnecessary, but it's too risky to remove
it.)

Change-Id: I8caa3dd945692a4ec0a1128199237de6b77aa719
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609117
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-09-04 17:48:17 +00:00
Jonathan Amsterdam f5afe0245f several: use stdlib functions
Replace some functions with simpler code that uses
recent functions added to the standard library.

Change-Id: Ifcd570679a897fb165a4307c7ce9f0f53fccea05
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609116
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
2024-09-04 12:20:54 +00:00
Jonathan Amsterdam 6851d87f22 go.mod: upgrade to go 1.23
The go.mod major version now matches what we use for Dockerfiles.

The version of staticcheck that works with Go 1.23 revealed several
uses of deprecated functions. Some of these were easy to change,
but other require more care and will be addressed in some later CLs.

Change-Id: I7f8344e419b8be6351d51db90faa64cef04057a0
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/609142
Reviewed-by: Robert Findley <rfindley@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-09-04 11:03:35 +00:00
Jonathan Amsterdam 676c19eae9 all: get all.bash to work
- Fix npm vulns.
- Fix the minimized-css check to look only at the working dir.

Change-Id: Iecf2c5ef35813fef16df22147f795cda814fef2b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/608037
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-08-24 13:58:12 +00:00
Jonathan Amsterdam fca1531c8b internal/godoc/dochtml/internal/render: move tests to txtar
Move most of the cases of the formatDocHTML test to txtar files.
They were too hard to read.

The remaining cases have to do with passing an ast.Decl.
We may move them later or we may try a different approach
to generating unique heading IDs, which is the purpose of passing
in the decl.

Change-Id: I99c71ed1f04b469cb540ff72a31a39eb6cbb1050
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/607616
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
Reviewed-by: Robert Findley <rfindley@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-08-22 15:27:54 +00:00
Jonathan Amsterdam 7be9f26791 internal/godoc/dochtml: add comments
Also change renderInfo's first return value to template.FuncMap
for clarity.

Change-Id: I2ad2906d260d405225d54723401276e419545ecc
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/607418
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-08-22 11:56:24 +00:00
Joseph Shin 53b9917759 internal/godoc/dochtml/internal/render: ensure unique heading ids
The existing implementation takes a simple approach of using the heading
text and replacing non-safe characters.  This leads to potential duplicate
heading ids, so use a map to keep track of used heading ids and generate
unique ids with various strategies until a unique id is generated.

Fixes golang/go#64582

Change-Id: I95f823c3782411454cff7353b49d90fab5799a45
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/573595
TryBot-Result: Gopher Robot <gobot@golang.org>
TryBot-Bypass: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2024-08-21 14:16:17 +00:00
Joseph Shin 71f98d56b5 internal/godoc/dochtml/internal/render: properly handle multi-line comments
The existing implementation rewinds lastOffset when indents are replaced
because lastOffset is calculated against the modified literal.  lastOffset
acts as a file pointer against the source so calculate offset against the
source literal before stripping indents.

Fixes golang/go#56762

Change-Id: I86fcef9eb24742e707b4a723a9767baac5433b9c
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/586099
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Hongxiang Jiang <hxjiang@golang.org>
Commit-Queue: Jonathan Amsterdam <jba@google.com>
TryBot-Bypass: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2024-08-20 21:09:57 +00:00
Tatiana Bradley 763ccd4c8d internal/vuln,static/frontend: display custom versions on vuln pages
On vulnerability entry pages, display "custom versions" (and a note
explainining them) if there are any "custom_ranges" published in the OSV.

(Once this is live, we will remove the disclaimer from the OSV details
since this will be redundant.)

Change-Id: I20c3b08ecb3f36ce0334553eef0dacd0fe57480b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/597616
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-07-15 15:20:26 +00:00
Tatiana Bradley f20ba84343 static/frontend/vuln,internal/osv: display info about unreviewed meaning on vuln page
For UNREVIEWED reports, display the word "Unreviewed" under the title,
and add a blurb to the feedback section explaining its meaning.

Currently, don't display anything special for a REVIEWED report. This is the
default.

Change-Id: I43ad7c324f6712a22a035c4aac725808cd50e7c3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/586101
Run-TryBot: Tatiana Bradley <tatianabradley@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-05-22 14:39:55 +00:00
Tatiana Bradley 41da83f650 internal/{frontend,vuln}: display affected modules
For vulns with no detailed package data, display affected
modules and versions.

Change-Id: Ibacdcd486cbb47b17a11d331692356a0603ac6d1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/581181
kokoro-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Run-TryBot: Tatiana Bradley <tatianabradley@google.com>
2024-04-24 21:17:54 +00:00
Tatiana Bradley 60fcee760b internal/osv,static/frontend/vuln: add a chip for standard library vulns
Change-Id: I0cb26136ddd58afbb53312dbd93f3ca1a5b8388f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/581176
Run-TryBot: Tatiana Bradley <tatianabradley@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2024-04-24 21:17:33 +00:00
cuishuang 6631f5443b all: fix function names in comment
Change-Id: I25ba419d2fc5878e14cf3b5026724e17efaaa39d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/579177
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: qiu laidongfeng2 <2645477756@qq.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@golang.org>
2024-04-16 22:18:14 +00:00
Jonathan Amsterdam 29d7d25705 frontend: display unexported vulnerable symbols
If all vulnerable symbols in a package are unexported, display
the unexported symbols instead of displaying "all symbols".

Change-Id: I509ac158670ed2529bc040abbd80e96bdde676a4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/571320
TryBot-Bypass: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-04-10 16:53:26 +00:00
Robert Findley b8abe08197 internal/postgres: exclusions should be case insensitive
Update exclusion logic to be case insensitive. When there is no go.mod
file, GitHub module paths may be spelled with any casing.

As a result of this change, we will no longer be able to exclude only
certain capitalizations of a module.

Change-Id: I67342cf3543e795e7795aa74081541d7e6fed18f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/576595
Reviewed-by: Jonathan Amsterdam <jba@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Auto-Submit: Robert Findley <rfindley@google.com>
2024-04-05 14:29:09 +00:00
Jonathan Amsterdam ea43129276 worker: minor improvements to main page
Move experiment section to bottom (it is relatively uninteresting).

Sort requests by age.

Change-Id: I7c0df9a9bfa4baa7b2960c5008c2e9ebc6a9d6ab
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568240
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
2024-03-05 19:44:12 +00:00
Jonathan Amsterdam acdeec62f9 worker: display request state on main page
Add a RequestInfo.State field to record the current state of a request
for display.

Add a function internal.RequestState to for updating the state,
and use it for long-running requests.

Display the state on the worker main page.

Change-Id: I1a04036982aba23cf27776fac75248972a1336f3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568515
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-03-05 19:44:05 +00:00
Jonathan Amsterdam df601fe449 internal/middleware,etc.: store requests
This CL provides several improvements to the worker home page:

- Display all requests, not just fetches.

- Link to the logs for each request.

- Link that will cancel a request.

At the heart of these changes is a new piece of middleware that tracks
all active requests, along with their trace ID and a function that
can be used to cancel them.

This change also affects logging, because the logger doesn't need to
maintain its own trace ID.

Change-Id: Id022170073d2d7ca4e45aaa1d78b216d8a512f35
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568236
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-03-01 15:36:33 +00:00
Jonathan Amsterdam d223bbd702 internal/worker: improve template-parsing code
We can turn some repeated code into a loop by using a function
that can create a TrustedSource from a dynamic string.

Change-Id: Id1bed179bfe3cf22585e44f03482f385a0163f06
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/568235
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-29 20:24:40 +00:00
Jonathan Amsterdam 31c5c2a273 internal/postgres: speed up imported-by counts
The SQL statement that updated imported-by counts was using an unindexed
column (package_path) in search_documents, so it ran very slowly (5
minutes to update 1000 rows, for example.)

Switching to an indexed column (package_path_id) should make it faster.

Change-Id: I8a32aff6ed43870af4508c382e3d9dabcec3155d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/567137
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
2024-02-27 17:34:43 +00:00
Jonathan Amsterdam f5332ac2de internal/worker: improve home and debug pages
- Move excluded prefix list to a separate page.
- Move debug pages under "/debug".
  (Previously the dcensus ones weren't even exposed.)
- Add links to debug pages at top of home page.

Change-Id: If26e139984b34cb6b2ef78af8ef154d96436a26a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/566418
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
2024-02-27 12:03:34 +00:00
Jonathan Amsterdam 70d3c43e75 internal/frontend,static/frontend: remove styleguide
There was once a plan to serve a styleguide for the formatting of
this site, but it never happened.

Change-Id: Ibd15abe9e3a16abc8cedde1bee2751f334abdbe9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/566419
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-26 16:35:55 +00:00
Jonathan Amsterdam 9b2cbf616e devtools/cmd/dumpdoc: tool for writing out package documentation
This is a tool for an experiment.
It reads package documentation from the database and writes it
to a file in gob format.

Change-Id: I3b6cfde24ce155a647c3c43caff68be63832f80f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/566416
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-23 17:34:38 +00:00
Jonathan Amsterdam 094b90f1e8 internal/worker: add batch param to update-imported-by-count route
Inserting 5000 imported-by counts into search_documents timed out.
Make that size configurable, so we can experiment with something
smaller.

Change-Id: Ie0c5f42a28c0468c600af6d836edc345529b769f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565682
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-22 22:32:45 +00:00
Jonathan Amsterdam 463d7c943f internal/postgres: support module@version exclusions
Allow a specific module@version to be excluded. Previously, only entire
module paths or path prefixes (at every version) could be excluded.

This will allow us to exclude specific versions that we can't process,
because they are too large for instance.

A necessary part of the change is adding a version argument to
the IsExcluded method. While we are there, we drop the error return
value, since it's always nil.

Change-Id: I68fd1c8f66ee654e56bd7dcf230a88959b32bbc8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565815
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-21 20:50:14 +00:00
Jonathan Amsterdam 375600cd10 internal/fetch: treat go.mod parse failure as BadModule
We were returning a 500 if we couldn't parse a module's go.mod file.
Instead return 490 (bad module).

This fixes github.com/AlCutter/crucible@v0.0.0-20240212163456-5ee0456321b3,
and others.

Change-Id: Ia65a4523c44e6baa65760cef07c434b32f80c678
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565678
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-21 20:22:16 +00:00
Jonathan Amsterdam 424156b7a0 internal/godoc: increase max imports per package
This will let us process git.runstone.com/DataCollect-BD/cisco-network-telemetry-proto@v1.1.0.

Also, remove an unused constant from internal/fetch.

Change-Id: Ib57078b84f5ee32b59fe934d153ede47d238e1d5
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565677
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
2024-02-21 20:22:07 +00:00
Jonathan Amsterdam e18d7814b3 internal/worker: fetch returns 404 on m@v
We were parsing a fetch path of module@v1.2.3, but returning a version
of "1.2.3", not "v1.2.3".

But we shouldn't accept this syntax at all, only the proxy syntax
"module/@v/v1.2.3".

Make that behavior change, but also improve both parseModulePathAndVersion
and its test.

Change-Id: I5e085b4845a939458838ca6f6da6600c854ed9d4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/565676
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-21 20:21:59 +00:00
Jonathan Amsterdam 71f709ab2d internal/stdlib: remove support for special branches
Remove support for the dev.boringcrypto and dev.fuzz branches.
They are no longer relevant.

For golang/go#65205.

Change-Id: If0e7060b148fe3ebff9139a6ce2f418b928a0bd6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/557415
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
2024-02-21 14:58:05 +00:00
Jonathan Amsterdam fa961d1241 internal/fetch: handle error in root package
If a module had a problem with the package at its root, we would
end up with two packageVersionStates. That caused a DB error
in upsertPackageVersionStates.

Distinguish between the module unit and package units in a couple
of places. That addresses the root cause of the problem.

Also, fix an unrelated bug that was causing a panic when we
tried to send the error to the errorreporting service.

Change-Id: Ia02faea50f4477999c6c6eca0674b8dde0f15bb3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/564076
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-02-14 17:07:49 +00:00
Michael Matloob 8984be28e8 internal/fetch: close directory file in defer in extractReadme
This addresses bcmills's comment in CL 559615

Fixes golang/go#65545.

Change-Id: I02a4ba48eb897f33b4f4c47911e9cc80c99f7869
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/562176
TryBot-Result: Gopher Robot <gobot@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Run-TryBot: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
2024-02-06 21:01:53 +00:00
Michael Matloob 299e4eeccd internal/fetch: remove comment that is no longer relevant.
It doesn't matter for the same reasons when we set HasGoMod.

Change-Id: I509233be3d85d2c5a4c948238ee26907e72faa3f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/562175
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Michael Matloob <matloob@golang.org>
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2024-02-06 18:25:40 +00:00
Michael Matloob 13788923a5 internal/fetch: split FetchModule in two
First we get a LazyModule that essentially just contains the UnitMetas
for each of the modules and then we process each Unit as needed so
that FetchDataSource doesn't need to process all the Units when it
just needs one. To construct at FetchResult, FetchModule will now get
a LazyModule and then compute all the Units.

A compromise to get this to work is that FetchDataSource will no
longer populate Synopsis, IsRedistributable and Licenses on
Subdirectories to avoid computing all those fields for all units when
returning a single unit.

MainVersion and MasterVersion are removed from FetchResult because
they were only set on the internal/worker.FetchTask struct that embeds
a FetchResult.
 
Change-Id: Ia0db850ae570d421712ec484ee8b7815a779128e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/557818
Run-TryBot: Michael Matloob <matloob@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
2024-02-06 18:18:13 +00:00
Michael Matloob f26e1ab5cf internal/fetch: split extractReadme from extractReadmes
In CL 557818 we're going to process readmes a single unit at a time,
so split a function that extracts a single readme at a time out of
extractReadmes so we can use that in the cl.

Change-Id: I5c39a95869de12e6f4aeebd2f42d87f24e147786
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/559615
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2024-01-31 19:21:10 +00:00
Michael Matloob c7eaeb891e internal: move UnitMeta.IsRedistributable to Unit
Change-Id: Ia0a18df32d752f20b0400b47860638c7d0910eb7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/556435
Reviewed-by: Jonathan Amsterdam <jba@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
2024-01-23 15:28:52 +00:00
Michael Matloob 506adfeab1 internal: move UnitMeta.Licenses to Unit
Change-Id: If2b3c1acd4830948e2d134517608f89771a7abcb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/555735
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
2024-01-22 17:11:06 +00:00
Michael Matloob 8b25f917fd cmd/pkgsite: fix case where we don't have runtime.GOROOT()
If pkgsite is built with -trimpath, runtime.GOROOT() returns an empty
string, so we don't know where the local go stdlib is. In that case,
don't try to serve a local stdlib and instead fall back to the
stdlibZipModuleGetter. Also fix two issues that show up when using the
stdlibZipModuleGetter. First, we try to prefetch fetch.LocalVersion of
the standard library when the server starts. Instead, prefetch latest
which will have the correct behavior with both the stdlib zips and
local stdlib. Second, when we're trying to fetch the stdlib using git
make sure that we're on another branch than the one we're trying to
fetch into because git won't let us fetch into the branch we're on.

Fixes golang/go#64903

Change-Id: I9dfb22d50b7738080490ce2682e0cf187c16d2d1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/554195
TryBot-Bypass: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-01-08 21:04:35 +00:00
Michael Matloob a0bbdd60d9 internal/tests/deps: skip test if we can't run go command
For golang/go#64942

Change-Id: Idffad578d4bb766dd5c9d8d1a500b8f68598b664
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/553695
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
2024-01-03 19:55:40 +00:00
Michael Matloob eb2fcc3790 internal/tests/deps: add more information to error message
For golang/go#64807

Change-Id: Ie2b95ccf0c77ceb8d4a138a767937e3751db7a76
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/553516
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: Michael Matloob <matloob@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
2024-01-03 19:47:16 +00:00
Michael Matloob 254ae6f648 cmd/pkgsite,internal/fetch: improve startup experience
When a user tries to look at the documentation for a unit for the
first time, we process all the packages in the module. Kick off
goroutines to do that so that the modules' packages might be processed
by the time the user sees them. Also parallelize processing the
packages in the module. That brings processing std from about 8 seconds
to about 3 seconds.

Change-Id: I505979d682897d2d1e55ce9e123e3ff4d1fb0e70
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/551776
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2023-12-21 16:03:52 +00:00
Michael Matloob 4356cead6c internal/tests: add a dependency test for cmd/pkgsite
Ensuring that it keeps its limited set of dependencies.
The test is rather slow: it takes about 5 seconds for me. I tried
parallelizing it and only got it down to about 3 seconds so I figured
it wasn't worth it.

For golang/go#61399

Change-Id: I34b75a1f77480e02368ddb7d552e89ab106bf28e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/551515
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2023-12-19 21:19:50 +00:00
Michael Matloob c63424d1f1 internal: move templatecheck tests to their own package
The new package the templatecheck tests are in is not a transitive
dependency of cmd/pkgsite, so that cmd/pkgsite no longer has a
transitive test dependency on github.com/jba/templatecheck.

To make this work we had to expose the templates from the
internal/frontend and internal/godoc packages for the tests to use.

For golang/go#61399

Change-Id: I1290ec24b53af77a82671c8fc068867e12857ce3
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/550936
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: Michael Matloob <matloob@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
kokoro-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2023-12-19 20:48:54 +00:00
Michael Matloob b2a01f8bdf internal/proxy/proxytest: use standard transport
proxytest is just used for tests so we don't need opencensus. Remove
the transport and the dependency.

For golang/go#61399

Change-Id: If2ef831deb098c393e625fdc288a8f6015540b48
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/550875
Reviewed-by: Jonathan Amsterdam <jba@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
TryBot-Bypass: Michael Matloob <matloob@golang.org>
2023-12-18 22:12:28 +00:00
Michael Matloob 8d0563ae24 internal/frontend: remove use of goldmark for readme rendering
delete the code that uses goldmark and clean up the variants of the
code that use the markdown parser

For golang/go#61399

Change-Id: I03e8c303086110278dd0f3994ba97729e0cbf7c1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/550039
Reviewed-by: Jonathan Amsterdam <jba@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
TryBot-Bypass: Michael Matloob <matloob@golang.org>
2023-12-18 22:12:03 +00:00
Michael Matloob 9ad9de49cb internal/frontend: generate https autolinks by default
This is a departure in behavior from the old markdown parser but it's
probably better to try https by default.

Change-Id: I0f06eda5b48e87200cd72a95daa08f54ce932749
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/550295
kokoro-CI: kokoro <noreply+kokoro@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
2023-12-18 19:36:43 +00:00