Store the contents of all valid license filenames, even if they don't
match.
Fixes b/144581896.
Change-Id: If4620d67525291eef2de2a81dc5875046b1d3dbb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/603790
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Now that completion data is being populated by the ETL, we can use it to
provide a JSON API for search autocompletions at /autocomplete?q=query.
This API reads lexical completions from Redis, sorts according to a
scoring heuristic, and returns the results as a JSON array.
For testing purposes, the pathCompletions function is moved from the etl
package to the complete package, and exported.
Updates b/143370178
Change-Id: I61d75296399222fa9ebfe8aadc909bf048389a23
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602064
Reviewed-by: Jonathan Amsterdam <jba@google.com>
We're going to compute auto-completions using redis sorted sets in the
'HA' redis instance (because we don't want them to be evicted by
caching). This data need not be very fresh, and so can be computed by
the ETL every so often, perhaps once a day.
This CL wires the HA redis instance into the ETL, and adds an action to
update these Redis sorted sets. Doing this takes 1-2 minutes and
consumes quite a lot of memory, so there are some optimizations used:
+ the underlying *sql.DB is used for streaming search documents, rather
than a method on postgres.DB. There is explanation for this in the
code, and additionally I don't think this is so bad as long as it is
only done in the ETL.
+ Completions data is loaded into temporary Redis keys and then renamed
to overwrite the keys that will be used by the frontend, in order to
minimize disruption while the completion data is being updated.
Updates b/143370178
Change-Id: I48caf77eb26d8ea7b60c2e117481bfa98ef089ca
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602062
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Avoid the use of the naked term "version" in the details
header. Instead, we have:
- DisplayVersion: the version formatted for human readability.
- LinkVersion: the version to use in links to the discovery site.
Also, use the link version in nav tab links:
Fixes b/144873325.
And improve test coverage.
Change-Id: Ida80867b6bd70a32c18f45b21b0b361b9e8e904f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/603789
Reviewed-by: Robert Findley <rfindley@google.com>
These tests had some deliberate errors and omissions to match bugs in
prod. Now that prod is fixed, we can fix the tests too.
Change-Id: I4e738f624701fb8ba421cf121e4db9a4e54ebd57
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/603241
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
We were using the formatted version as one input to our latest-version
check, and comparing it with the linkable version. That fails with
pseudoversions.
Use the linkable version in both cases.
Fixes b/144860268.
Change-Id: I0f4c82dbec42863378e5d649bbbe5ebe99de8b53
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602829
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Store the output of licensecheck.Cover in the licenses table.
This will make our lives easier if we ever want to revisit our
coverage parameters.
In the process, generalize scanning of jsonb columns
into Go values.
Change-Id: Iadc89c89c2590e0894fefcdcd461283d90b2e701
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/601900
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
If the request URL is unversioned, the module link on the overview
page should be too.
Updates b/144217401.
Change-Id: I4d10555639a1ffd44948206694b2cea3cb2a036b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/601915
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Some functions no longer access the DB. Remove the context
and DB args, and rename from "fetchFoo" to "constructFoo".
Change-Id: I4124ba8f3f2633550f3c6470e25050aa12a67162
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/601913
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
We're going to store completion entries in redis sorted sets, which
means they need to be string-encodable and string-decodable, in such a
way that they preserve their lexical sorting. This CL adds a 'complete'
package with a 'Completion' type that implements such an encoding, so
that it can be shared by both ETL and frontend.
There are also some constants added that will be used in later CLs.
Updates b/143370178
Change-Id: I94700d8f4e5deab4d74e9ef47cc8b124087f7275
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/602061
Reviewed-by: Jonathan Amsterdam <jba@google.com>
A recent change (not yet in prod) introduced a bug:
On the overview page of a module with a pseudoversion, the
module link has the formatted version string (e.g. "v0.0.0 (201909091-392039)")
instead of the actual version.
Fixed, and added server tests to check.
Change-Id: I9f88272e45f496f21d741a3f119396c7223c7ee9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/601903
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
By storing the output of the licensecheck package in the DB, we
can easily recompute whether a license is redistributable if
our criteria change.
Change-Id: I07634b96e469e6d4bfda362624999aee5ff1848b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/601142
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Add data-test-id attributes to the license and module parts
of the info label, so we can more reliably extract them
for tests.
This practice is suggested by the TotT "Select DOM Elements by User-Facing Properties"
Change-Id: Ie4bf1cac8c3e88b728c2561256d210ab58819968
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/600820
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Factor out common structural checks on pages, so we can
use them in more than one place.
- Add internal/testing/pagecheck for common checks on page contents
- Use in internal/frontend/server_test.go
Change-Id: I744cf090f6bfad8c788be5b21cb5ed80f9adf8de
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/600800
Reviewed-by: Julie Qiu <julieqiu@google.com>
Move the inner loop to a separate function to clarify control flow.
Updates b/144581896.
Change-Id: I42527d197153cba03734a616010bde968c48d575
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/601140
Reviewed-by: Robert Findley <rfindley@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Move the html-checking code from internal/frontend tests to its own package.
Ths is a first step towards making these structural tests usable elsewhere,
like in an integration test that runs against a live endpoint.
Since these are now exported, give them better names in some cases.
Also, generalize to use regexps where that makes sense.
Change-Id: I1a3e0c09954d9cb07200c087db17e499a9d0d9a7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599089
Reviewed-by: Julie Qiu <julieqiu@google.com>
The code for computing the formattedVersion was moved to a function,
which is now used throughout the internal/frontend package.
Change-Id: Ib7f1b9dc658a966094afbda50c4ecd886bd4190c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/596566
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Use server-side string replacement to determine what to show for the
latest-version badge, rather than client-side code that calls back to
the server.
Using the same technique as for script nonces, we embed placeholders
in the HTML, and a middleware layer replaces them with the correct
values. The HTML with placeholders is indefinitely cacheable,
yet we still serve a page whose body can change dynamically.
The replaced information is kept to a minimum: the latest version for
use in the "go to latest" link, and a CSS class. Both alternatives for
the badge are in the HTML, and the CSS class determines which one is
displayed.
An additional challenge we did not face with nonces: how to extract
information embedded on the page that we must use to determine the
placeholder values. In this case, that is the version being served,
the module path, and in the case of a package, the package path. We
could find that information by parsing the HTML, but that is overkill,
and slow.
I first tried embedding the information in an HTML comment and
used a regexp to extract it, but surprisingly, HTML comments are
stripped out by html/template, and there is no easy workaround. So I
added the information to the HTML as data-XXX attributes, but I still
extract them by regexp for speed. Since this data appears before any
user-provided text (like documentation or a README), and we find the
first match, there is no way for users to spoof it. But it is fragile,
as the comment in details.tmpl points out.
Now that the latest-version badge is done on the server, we can
add checks for it to TestServer, which I've done.
Fixes b/144098197.
Change-Id: I3a126f1cac48cd35c0317d47035bf227483cdbf1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599082
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
- Make the highlight color match the sticky header. This also
makes it pass the AA requirements for contrast, which the previous
color did not.
- Add some padding to items so the highlight is not at the left edge
of the item text.
- Make the text input bigger.
Fixes b/144358048.
Change-Id: Id484afbf66f0973cf7e39a67efd5cee0a6e933f4
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599083
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
Browser tabs now read "foo package" or "foo module" or "foo directory",
so they are distinguishable when space in the tab bar is tight.
Fixes b/144278455.
Change-Id: Ia85c8fcfb4326a509aa2cdb27ce5ceea91f89642
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599085
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Some READMEs link to themselves with fragment-only links, like "#getting-started".
- Make blackfriday generate `id` attributes for headers.
- Don't translate fragment-only links.
Fixes b/144354397.
Change-Id: I78c50d794aca1498bee2bdc55cd1e18cf4ac0a15
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599086
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Put all test-related packages there (sample, testhelper, integration).
production risk: none
Change-Id: I06673aaa5c3fd6966410f1c1ad2b34e4f2d88f94
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/599080
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Handle the idiosyncratic mapping of go1 to v1.0.0.
The following frontend paths changed behavior:
- std@go1?tab=versions and archive/tar@go1?tab=versions both work now
- std@go1.0 and archive/tar@go1.0 now return 4xx
Fixes b/144364761.
Change-Id: I2345faeedb1e418028422feaf7d9ad0233bc08ff
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/597504
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Fix the position of the latest-version badge.
- Fix the height of the DetailsHeader-title div.
- Inline the copy icon.
Also, rewrote the breadcrumb test to use
the new html-scraping stuff, to make it less brittle.
Tested:
- Using "Slow 3G" mode in Chrome.
- I made Heschi look at it and he said it's fine.
Change-Id: I3706a6e59e70c7094a2c7fd1d210dabf01a51cb3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/597501
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
Previously, the TestServer function fetched pages and treated the HTML
body as a string, looking for literal substrings. With this CL, it
parses the HTML and looks for patterns in the structure.
This makes the tests more precise and more flexible. We can still
get the old behavior by writing
in("", text("something"))
(which means: in the entire body, look for the string "something").
But we can also, for instance, test which tab is selected, by writing
in("li.selected", text("Imports"))
The previous tests were brittle in that small, irrelevant changes could break them.
For example, a test for the literal string
<a href="foo">bar</a>
would break if, say, a class were added to the tag:
<a class="c" href="foo">bar</a>
Now we can ignore other attributes by writing
in("a", href("foo"), text("bar"))
One thing we lose is the ability to say that one string must occur
after another; the old list of strings had to occur in the order they
appeared. But that is more than made up for by the ability to focus on
a particular element or subtree when looking for a string.
This CL also fixes a bug in the test where the module URL was not
being checked.
PRODUCTION RISK: none
Only tests are changed.
Change-Id: I8304c279df25f32a4a2b669a6bb6a17fa7baef44
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/597500
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
GOOS and GOARCH are now only displayed for packages where GOOS != linux.
The copy is changed to Documentation was rendered with GOOS={{.GOOS}}
and GOARCH={{.GOARCH}}.
Fixes b/144172566
Change-Id: I28c06227a5a86f23a564a5014105588e7d454fb0
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/597563
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
This reverts commit 30f40336360fd7860b59a6fe7526465b522ace38.
Reason for revert: Testing is done, so this handler is no longer needed.
Change-Id: I13d18509bc50571ac740215fd37d76cb250f7379
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/597566
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Before launch, I just want to sanity check our non-trivial alerts by
forcing errors. Rather than overload our database as I have done in the
past (which is not something I want to do the day before launch), I'd
just like to add a handler that allows me to specify an error code to
trigger.
I'll roll this back after manual testing.
Change-Id: Ia8b21a13617c0e48ebaabd1dae2c0e8ba68a6a78
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/596571
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Also cleans up some unused CSS.
Fixes b/144213173
Change-Id: Iecb50b9e9fd6aff05530fe890d51f81a271e04ab
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/596740
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
Do not count package A as importing package B if they
are in the same module.
Change-Id: Ic0f98af8d72b8fff29e105779c6eb619214335e9
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/596525
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
These are now being handled by the go.dev static app.
Change-Id: I3831465bc8d0e1518bdb9bb29be5f7192efbc7d4
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/596644
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Due to some refactoring, it was in the wrong place.
Change-Id: I2d9632c27e0716d1677c4b1a1a2604f7e283dfeb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/596521
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Jonathan Amsterdam
Rob Findley
Julie Qiu
Andrew Bonventre
Alexander Rakoczy
Andrew Bonventre