Avoid copying a string if it's already valid unicode.
This saves memory, which may matter. See bug for context.
Updates b/158067614.
Change-Id: I402b7823cba240e593fe1b094f4f85a9cf5eeec2
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/762241
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Make it possible to add a label to a context, so it appears in all log messages.
- Add the module and version as a label when fetching.
This will make it easy to find all log messages related to the fetch
of a single module.
Change-Id: I6c564b6aa58a494180d61168e739ef62dd67a64c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/760101
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Previously, setting an experiment with the experiment query param
required the flag to be present in the experiments table.
Feature flags can now be set with the experiment query param without
first being added to the database.
Change-Id: Ia24862f2e05414547b5f2dfb96561af49b568835
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/760279
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Andrew Bonventre <andybons@google.com>
This change implements the beginnings of a sticky navigation
component on documentation pages.
It is only implemented for wide (desktop) viewports and has no
JavaScript to show selection while scrolling. That will come
in later changes as the hierarchy gets fleshed out.
This also slightly adjusts the amount of space given between
the site header and an element scrolled into view by an anchor.
Updates b/148095016
Change-Id: Id213f1dcb7a2600d5c64bf955c79e0bbc55581db
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/745552
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reorganize the code of saveModule so that all changes that assume
we have the latest version happen at the end.
Change-Id: I53b261f1fc24bd48001542b4f0e1d6bbc5f75d29
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/760099
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
- Add an arg to Transact for the isolation level
- Remove TransactSerializable
This makes it possible to use other levels, and makes it easier to see
which level is being used for each transaction.
Change-Id: Iba5e2920b4139e5e2f0f8c6b331a658d7c84f60f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/758942
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Rather than reading experiments from the context, the experiment set for
an InMemoryQueue is now pass in as an argument. This makes it more
explicit what experiments are being set.
Change-Id: Ib68f567ea5b7ff0fc2157ad2713c76d33827c442
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/759927
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
At the moment, taskIDChangeInterval is a hardcoded value in
internal/queue. However, we will soon have two task queues running,
which require different change intervals, so this value is now set in
internal/config.
Additionally, the taskIDChangeInterval for the worker is changed to 3
hours.
Change-Id: I498abefce6543005463be7da99a5a778f3a6e973
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/758919
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
We've seen documentation that is not valid UTF-8
(github.com/xuender/oil/math). That results in a DB error when written
to a TEXT column. Make the documentation valid before writing to the
DB.
Change-Id: Ib2ccdd1dfb3b66af9d444b81216f561a8d906d8a
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/757995
Reviewed-by: Julie Qiu <julieqiu@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
./all.bash prettier now runs on files with a .md file extension. Several
existing .md files are formatted.
Change-Id: Id083a932cdd5db105ae7494b71bcad340b080132
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/756682
Reviewed-by: Andrew Bonventre <andybons@google.com>
The contents of this directory are not being used.
Change-Id: I0660f09ba8bd481afa806ad67b0c889d8f055ec7
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/756689
Reviewed-by: Jonathan Amsterdam <jba@google.com>
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
A .prettierignore is added so that content/static/js/*.min.js files are
ignored when running prettier.
Change-Id: I13738e03a1681475dd3d767258aefcaec73c99d5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/756683
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Running with `go test -record` revealed some 404s (pages that used to
exist but no longer do). I replaced them with similar pages.
Change-Id: I6903229e8221923e22338bdae3f112308aecd19e
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/756267
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Our requeue logic is such that large modules (ones with > 1500 packages)
are requeued last, since they can take a long time to reprocess.
However, with large modules, it's possible that the modules that are
queued do not finish processing before the dedupe period for Cloud Tasks
is up, causing the queue to be backed-up with duplicate modules.
A limit of is now added so that at most 100 large modules can be added
to the queue at once.
Change-Id: I0d2a9b709b9a9972b609e9c3af125d58d37f4488
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/755584
Reviewed-by: Jonathan Amsterdam <jba@google.com>
At the moment, all errors for servePackagePageNew will return a 500,
because of derrors.Wrap. This is fixed by checking to see if the
err passed to serveError is of type serverError, and if so, return the
status code of the underlying serverError. Regardless of the error
type, the full error will be logged.
Change-Id: I502b174e1efb9f856f17ffb3614fedc909ef162c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/754806
Reviewed-by: Jonathan Amsterdam <jba@google.com>
A teeproxy service is add to tee traffic from godoc.org.
23683d71bb
These requests are used to collect data about requests to godoc.org, and
ensure that all incoming traffic to godoc.org can be handled
appropriately by pkg.go.dev.
Updates b/157405502
Change-Id: Ia696457a3867ba0cc883c6ebfd38b45d671e160f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/754075
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
At the moment, experiment flags won't be set after they are passed from
a service to the InMemoryQueue. Experiments with a rollout > 0 are now
set when initiating the InMemoryQueue for local development with
cmd/frontend and cmd/worker.
Change-Id: I209024e8281b735d37b4e76d81b4b05ff500e7c3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/753601
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Record metrics for requests that are allowed because they are on the
list of accepted URLs.
Change-Id: I25d9f72cb92430854a8903444a21bc64785941b1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/754550
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
+ Update the title suffix from go.dev to pkg.go.dev and some
incorrect usages
+ Add a meta description for SEO purposes
+ Lazy-load images in the footer (requires setting heights and
widths for both images because otherwise loading could cause
reflow
(see https://web.dev/native-lazy-loading/)
+ Use rel=noopener in places where it is not present or
where it was redundant with noreferrer. Both are not needed
(see https://web.dev/external-anchors-use-rel-noopener/)
Change-Id: I46767bd67e6db01c4d5921926381d5b32a89f018
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/752824
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
+ Load all JavaScript files at the end of the page so that DOM
parsing won’t be blocked by script execution.
+ Extract out Google Tag Manager script tag and place init logic in
a separate file instead of using an inscrutable, minified inline
script.
+ Add the JavaScript compilation script.
+ Update license headers to have a range so that multiple licenses
are not included in the minified source when only the date is
different.
+ Update the prettier config to avoid arrow parentheses unless they
are needed to match existing style.
Change-Id: I7331ee7fab04eb2ff45b5f59928b3e5817d35813
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/752822
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Reviewed-by: Alexander Rakoczy <rakoczy@google.com>
Wrap DB errors in BulkInsert and BulkUpdate, instead of hiding them.
This will allow TransactSerializable to retry the transaction; now it
is just failing.
Also, log the number of retries so we can get a sense of how congested
we are.
Change-Id: Ice538c5c4de56d88c89ec7a290767c0d5c0b9e39
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/752351
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
There was a bug in the requeue logic that caused all modules to be
requeued when next_processed_at < NOW. This is now fixed so that only
module version states with status=0 or status >= 500 is picked up.
Fixes b/157403463
Change-Id: Ieffde87362c2cd05391780a20dba9647884c99a8
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/754074
Reviewed-by: Jonathan Amsterdam <jba@google.com>
It consists of three separate statements which should happen together.
Change-Id: I8a51dc34fd85e701996bc490932840052bd31162
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/752310
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
If an experiment flag is set, use a serializable transaction
to insert a module.
Change-Id: Ieca2d5ce68f456d48e69b9de6fc3714a130cbca3
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750955
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Use the context passed to Transact for the transaction.
Previously, we used the background context, because that is what
sql.DB.Begin does. But we should use the argument context.
This is a behavior change, but it should be unnoticeable. All our
transaction functions use the same context we pass to Transact, so the
only difference this CL makes is that the DB calls in Transact itself,
like BeginTx, Commit and Rollback, now have the potential to fail
because the context is done. In practice that shouldn't matter.
Change-Id: I5eeff51bcd28cb877e5288b83d9e08fa3d249482
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750953
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add DB.TransactSerializable, which executes a transaction with
serializable isolation. Although serializable transactions are more
expensive, they reduce the risk for anomalies, like the constraint
violations we sometimes see.
This CL does not change any existing behavior. In particular, the
DB.Transact method, despite calling sql.DB.BeginTx instead of
sql.DB.Begin, uses the background context exactly as Begin does
(source:
https://github.com/golang/go/blob/go1.14.3/src/database/sql/sql.go#L1689).
Change-Id: Iab9c99ca35de9ef884b149a3b39ecac3300ecac0
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750952
Reviewed-by: Julie Qiu <julieqiu@google.com>
This change removes Google Analytics in favor of just Google
Tag Manager since you can use the latter to send page view data
to the former.
Additionally, adds Google Tag Manager’s debugging view CSS domain
to the content security policy.
Only renders GTM resources when not in development mode.
Change-Id: I54149c378f7226e24d836c730a8fed641d9bcb54
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/751128
Reviewed-by: Julie Qiu <julieqiu@google.com>
This change replaces the reloadTemplates field with a more general
devMode that will allow for changes to ease ergonomics when
developing locally.
Change-Id: I87132830f51070cfc3905cf739d87ddc35f2f42b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/751682
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Somehow we lost the error return from the Transact call in saveModule.
This restores it.
Change-Id: I5d0f2d86ccc78a7506307e1d13a475930e36afa5
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750954
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
It's more idiomatic in Go to use nil rather than an empty slice, in
situations where either works.
Change-Id: I1f85ba39772b873b7b35ca35837fecddd3c33ba0
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750950
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Now that we are using basic scaling, we can use a longer timeout for
all requests, including fetches. So the special fetch timeout, and the
detached context and tracing that went with it, are no longer necessary.
We still enforce a timeout on requests via middleware.Timeout, using a value
that comes from GO_DISCOVERY_WORKER_TIMEOUT_MINUTES.
Updates b/141406364.
Change-Id: I39266a3f08b1f7e2fea3fdbec05bf092ea974d9b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750119
Reviewed-by: Julie Qiu <julieqiu@google.com>
Construct the sample SourceInfo from the given module path, not
the global one.
Change-Id: I2d61d0b234c591a2a6fb03bfbf329549cc0df507
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/750122
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
If the URL path doesn't match anything, but it could be in the
standard library (e.g. "json", "http"), then look for a unique
standard library path that ends with the given path, and redirect to
that.
Updates b/149933479.
Change-Id: I0d7c16620c1d0f7c8326302bac1c1423daccfd21
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/747788
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Create test servers, their handlers, and middleware in
a single function.
Also factor out adding experiments to a context.
Change-Id: I1a4046e1d190ed7c5f3fe3d12b999bc947821b5c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/747787
Reviewed-by: Julie Qiu <julieqiu@google.com>
Add GetStdlibPaths, which finds all paths in the standard library
that end in a given suffix.
The goal is to redirect stdlib paths like, e.g. "http" to "net/http".
Updates b/149933479.
Change-Id: Ib79d55ebb52f1203d8e68f52f6ed8c05db786dfa
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/747784
Reviewed-by: Julie Qiu <julieqiu@google.com>
Jonathan Amsterdam
Julie Qiu
Andrew Bonventre