The Count aggregation is the number of data points recorded. We
want the actual value.
Change-Id: I1141c731d1fef105cee9628e7b718a5f3573adde
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/323329
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
This reverts commit 61bac80a3d.
Reason for revert: if we queue in arrival order, then multiple versions of the same module that are published together (which seems to happen a lot) will be queued together, and will conflict with each other on the module path lock.
We'll have to remove the processing lag alert, but instead we'll alert on total backlog, which is more robust anyway.
Change-Id: Ia1f9f5c7ec8b0b6577527d3dd6060b692c7a7b28
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/323172
Trust: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Add a metric that records how many modules need to be processed (those
with status codes of 0 or 5xx). That will tell us how much the worker
is backed up.
Change-Id: I68c47255e3720b5f978037c1251b0e502961d089
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/323109
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Add a flag to gen_zip_signatures so we can check if a particular module
matches a signature we already know about. That will let us determine if
it's worth adding new signatures to the list.
Change-Id: I953587db71e41145f77692d1f61c3915d010d724
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322834
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Forks of github.com/ethereum/go-ethereum are clogging the worker queues.
For golang/go#46341
Change-Id: If41068ba32c6c7ab65b07eb238094ac9a845c55e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322833
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Adds the GO_DISCOVERY_E2E prefix for all e2e env vars.
The docker container for nodejs now gets the relevant
variables passed along in the docker run command.
Change-Id: I43ae4c52fee7be47c6f6afc9265d0d7e01d54015
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322831
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
I noticed some contention around the table lock that
upsertSymbolHistory was acquiring. (I looked at the logs of tasks that
had failed, and noticed that their last statement was the LOCK,
suggesting that they blocked on acquiring the lock and eventually
timed out.)
Instead of locking the table, we lock the module path, as we do
elsewhere for latest-version inserts. That should avoid conflicts in
upsertSymbolHistory and reduce contention, since two transactions will
only block each other if they are working on different versions of the
same module.
We acquire the lock earlier than we need to, to avoid passing
additional information down the call stack. That shouldn't matter
much.
Change-Id: I9c86e1ff6c638baade70fdee25e0ee31b30a03c4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321790
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
We have a pretty clear pattern for the basic snapshot
tests developed. Moving the common logic to a helper
function for better code reuse across tests.
Change-Id: I0a050587ce4a3ebbd4dd07d67381b6bcecbf3281
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322652
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
We will need to run npx commands with docker when
running tests in CI. This change creates an npx
command in all.bash and modifies the docker_nodejs
script to share networking with the host so it can
connect to a local instance of the frontend.
Change-Id: Id8f32a7ff5bf017cb55c531b1e37a3f8aab0e73a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322635
Trust: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
We are planning to add the basic snapshot tests
as a check in CI. Separating the search page test
because its content is not static and will
eventually require a more detailed test suite.
Change-Id: I02640c0051a08ed6eddda49aa47ce2a8938a11f4
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322634
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Add new columns to package_symbols for larger sized references to path ids.
After copying the existing data, we'll rename the column in another migration.
Change-Id: I3c528de27b6c6114e118683af629d9ee7412737e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322031
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Also, sort the map entries by module, version instead of signature,
so new module entries appear together.
For golang/go#46341
Change-Id: I0cfc95a9281fd11ae4ab52e4608b22663e0f8b54
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322530
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
If the worker sees a module without a go.mod file that is identical to
a one of a hardcoded list of known modules but has a different module
path, then the worker will mark it as an alternative module and avoid
processing it.
This implements the scheme described in the issue, except (1) we use
the AlternativeModule status code for this case instead of a new code,
and (2) we require that the versions match in addition to the zip
contents.
This first CL has the signatures for aws-sdk-go. We will add
more in subsequent CLs.
For golang/go#46341
Change-Id: I6d429464fcb48da43f9c874bc9b37049466bafb6
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322529
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
First of a series of migrations that will enlarge the types of columns
that refer to paths.id.
We'll do each table separately because they can take over an hour.
Change-Id: Iff254e0d8f93d3e3bd49685ea355d525e381eba1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322170
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Validate the constraints we established in the previous migration.
Change-Id: I3196bd9012ea20c32d2a0573c78c992d3ff212e8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321430
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Automatically sizing a textarea height to fit the content is
not possible without JavaScript. We get around this by
initially rendering the example code in a pre tag and then
replacing it with a textarea on page load.
Fixesgolang/go#46302
Change-Id: I580a09bf348cf16a3143e52fa253100e63e7d3ed
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321833
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
The fetch test is renamed to indicate that it is the 404 page of a
package that does not exist, not the loading page when a package is
being fetched.
Change-Id: Ie7cd082c5c0ff790016e8283af74332d447c1437
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321849
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Add a column to paths of type bigint, which will eventually
replace the current `id` column with a larger datatype.
Altering the type of the `id` column directly would
lock the table for a long time, so instead we'll do the following:
1. Create a new column `big_id`, and arrange for it to have the same
values as `id` for newly inserted rows. (This CL.)
2. Fill in the existing rows. This is just
UPDATE paths SET big_id = id WHERE big_id IS NULL
which takes under 10 minutes (tested in exp).
3. Alter all foreign-key constraints to point to big_id. Validating
the constraints takes a while, so alter them with the NOT VALID
clause and validate them later. See next CL.
4. Drop `id`, rename `big_id` to `id`, and make that column have the
right properties. (The four lines of
000084_new_documentation.up.sql starting on line 22 provide the
template.) This took a few seconds in exp.
Change-Id: If4e0a08c07a674a47da8448ac9bb7129ccbea298
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321012
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
That is what our frontend runs on.
Change-Id: Iae80f7141f5a88f76567cce21a115b3385fbb2b5
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/322009
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Since we clean pseudo-versions now, a test that depends on one may
fail in the future.
Replace the 'no documentation' case with a module with a tagged
version. That was the only test case that depended on a
pseudo-version.
Update the no-documentation goldens.
Also update the license-policy goldens. We just added an new license, which
broke those.
e2e tests now pass for me in staging and prod. Woot.
Change-Id: Ib0cb46a8a0e862f33ea1933df89847fb07fb9eb0
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321791
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
The content of the styleguide page is generated from
the markdown files contained in the component directories
of content/static. We use goldmark to parse the files
into html sections and generate an outline for the sidenav.
Goldmark is extended to render html code blocks as both
html and code snippets.
Change-Id: I023edb77114001439be58e00d5a48198181f165a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320654
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
TreeNavController is an improvement on the DocNavTreeController
that takes care of adding appropriate aria attributes to simplify
the html files, It adds accessibility attributes to a tree, focuses
the top most link for headings visible on the page, and implements
the WAI-ARIA Treeview Design Pattern with full keyboard support.
Change-Id: I2c0e188ca34854f068b7721d8a047721ce563bcb
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320650
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Adjusting the failure threshold to 0.1% will reduce the
amount of test flake and is more than enough to capture
the sort of breakages we care about.
Change-Id: If936a1e6929cf0df22666d54bbdadf1b1add49b8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321149
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
A newer version of goldmark supports parsing attributes
like class and id on markdown elements into html files.
Change-Id: Ieb9238d081598c27bef084ec212d55e7d0a10937
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/321073
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Updates prettier file glob to ignore JS files
and process TS files. The repo should not contain
any JS files outside of generated or minified code.
Change-Id: I1167b96c1d0d2ab2da5e9cf4d40c2b2fd79e70fd
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320990
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Data is now upserted into documentation_symbols, to fix a bug where the
package symbol ID in the database did not match that of the one that was
going to be inserted.
This was due to changes made to the package_symbols table in CL 315309
and CL 315310.
Change-Id: I1d26132a55339d1b747d8c2fb5dfa1cb0b153558
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320689
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The styles for the components in the guide will
make up the base styles for pkg.go.dev and eventually
go.dev. The markdown files provide examples of
use, will be rendered as sections in the styleguide
and are used in the JS tests when applicable.
Change-Id: I3743d53586114287c4537d0f737375c7dbc7a46d
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320572
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
We weren't reporting on NotFound, but the right code is NotFetched,
since that is what is returned when proxy fetch is disabled.
Change-Id: I554d5186fb4db0346fb9456fe9fc55091066abda
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320751
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Deadlock is rare but we have observed it when multiple versions of the
same module are inserted concurrently.
Change-Id: Idd28bd558009ff053ac198bdd310fa780af66f83
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320750
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Increase the maximum number of imports we allow a package to have.
The value was initially 1000, arbitrarily. But there is at least
one package (github.com/mgruener/iso20022tpl/internal/iso20022)
that exceeds the limit.
Change-Id: Ie9c7e449179ae2caad872a494a59fa49484b316a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320749
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
This change makes removing the title header from the
markdown TOC configurable. For the styleguide we need
to preserve document titles.
Change-Id: If43a1171f44637c98c15330ff1900c4f305511c2
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320653
Trust: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
The styleguide demonstrates usage of reusable components and
styles on pkg.go.dev.
Change-Id: I7aeb94a9bdc8b7fe12cc2832f8de0be0895ebd97
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320651
Trust: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Marked is used to parse markdown files in the JS tests
for the styleguide components.
Change-Id: I9b883b10a5f590977eb2c7f88ca019e1b1b20ed9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320571
Trust: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Importing a standard set of icons from the Google Material UI
and splitting image folders by domain for better ease of use
and discovery. A future CL implements a styleguide that relies
on this directory structure to generate sections within the guide.
Change-Id: Ibc08ff5a092d1d119a85fe7c4e44e8faeafe1512
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/320570
Trust: Jamal Carvalho <jamal@golang.org>
Run-TryBot: Jamal Carvalho <jamal@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Jonathan Amsterdam
Jamal Carvalho
Julie Qiu