Package the logic to read reports from a vulndb-like repo into
a Client, and update all callers.
For now, the behavior under the hood is basically the same as it was, but
the callers do not need to carry around the specific maps needed
to access certain information.
Change-Id: I7bd28bfbf2a40d5d2bc37453051f866517086f23
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/562177
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Use the new pkgsite client instead of making direct calls to pkgsite.
Change-Id: I8d7f107740679ea1e2475f44b666824b8548b8b0
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/554356
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Move existing logic to triage v4 CVEs to the cveutils package.
This will make it easier to add tests and implement triage for v5 CVEs.
Change-Id: I4872af391a33500dd7236795a910ad3a6998b5e0
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/550857
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Use the log/slog package instead of golang.org/x/event.
This removes x/event from the repo.
Until Go 1.22 comes out, use golang.org/x/exp/slog, because
we still test one version back (Go 1.20), and log/slog appeared
in Go 1.21.
Change-Id: Ia0f645069c778741ebb8c359161816a303a2175b
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/552675
TryBot-Result: Gopher Robot <gobot@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
Adds the ability to pull in fresh test data for v5 CVEs, and uses this
to add a test repo file containing v5 data.
For golang/go#49289
Change-Id: I36f516c75f2a1a241f614db7f5bb69555be096fc
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/545299
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Instead of exposing a global public variable, require that users of the
proxy package instantiate their own proxy client. This makes the package
harder to misuse, and makes it clearer where we are using a real client
vs. a test client.
Change-Id: I2f0f5895065e6efec1d50b3ac34cb04847e3b002
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/524457
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Plumb an explicit proxy client through as many functions as possible,
using the default client only in tests and top-level code.
This will allow us to identify and clean up tests that use real proxy
calls and should use mocks.
Change-Id: Ibd6423ea77c2007424c4539fe25f78c5b1f4764a
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/524135
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
No-op refactor to remove redundant words like "get" from function names.
Change-Id: I899556dbbe8caa790593bee8d952d1b3f557387b
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/497499
Run-TryBot: Tatiana Bradley <tatianabradley@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Removes code to scan modules for vulnerabilities, as it is now obsolete.
Change-Id: Ib5fe0f34f2d53709fe05db582e74e94b930b37a5
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/459836
Reviewed-by: Tatiana Bradley <tatiana@golang.org>
Run-TryBot: Tatiana Bradley <tatiana@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Instead of initializing a new client in every function call, move the
functions in internal/ghsa to methods on a client.
This will make it easier to add unit tests in a follow up CL.
Change-Id: Ifdd7ee04e884822a94d489d4f6fde3035441f152
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/458202
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Tatiana Bradley <tatiana@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Previously, issue.Client was an interface, so that we could use a fake
client for testing.
This made it difficult to jump to definition, when debugging issues with
the client. Additionally, the tests weren't very effective, since they
didn't traverse the githubClient code path at all.
Replace the test setup with a mock http server, and make issues.Client
to a struct.
Change-Id: I878846ce69c2979b45ff6e8494bc0f7c020055ea
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/458203
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Tatiana Bradley <tatiana@golang.org>
Whenever the worker files a new issue on the tracker for a GHSA or CVE, it includes cross references (shared CVEs, GHSAs, or modules exluding std and cmd) in the issue description.
Change-Id: I8c10e2f9835c7ddae7ad0427c219edc04a9a7ef6
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/453501
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Maceo Thompson <maceothompson@google.com>
Reviewed-by: Tatiana Bradley <tatiana@golang.org>
The worker was missing some GHSAs because it always filtered out GHSAs with CVEs (and sometimes CVEs are miscategorized as not Go vulns, aren't published yet, etc).
This change modifies the logic to look at all GHSAs and create an issue if there is not yet an issue for the associated CVE.
Note that this leaves a gap (which will be fixed in a subsequent CL) in which a CVE that is later found by the worker will have a duplicate issue created for it.
Change-Id: I54008c2b2772ee6de9ece2f129de8668e80bed27
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/432095
Run-TryBot: Tatiana Bradley <tatiana@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Use Go 1.18 to build and run the vuln worker.
Change-Id: I1dc24ce515feb8e22c5488ef4ae0f714a4495011
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/393838
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Normally, a module isn't scanned if there is a record of the scan
in the DB. Providing a "force" query parameter or command-line flag
overrides that.
Change-Id: I1d8836cd011060feb0ef2cf33e033abb2dbe9e67
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/393835
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Complete support for scanning modules.
- Add scan-modules command to cmd/worker.
- Remember modules we scanned in the Store.
- Track the last-modified vuln DB time to avoid unnecessary re-scanning.
Change-Id: Id2b6d3b2d91c6617d31f4fe6997babba2db220bd
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/393695
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
This is the first CL for a new task for the vuln worker: to scan
a selected set of modules for vulnerabilities.
Establish a new server endpoint, /scan-modules, to do that.
Currently visiting that endpoint scans the list of modules unconditionally.
A future CL will skip the scan if the vuln DB hasn't changed.
Hardcode a list of modules in the golang.org/x namespace.
Fetch each one from the proxy, and run vulncheck on it.
At present we just log any vulnerabilities we find. Later we'll
file issues to a GitHub repo.
Lastly, change the base image for the service to one that
has the go toolchain, since go/packages requires it.
Change-Id: I1de571d24d683b080542c5c40b55767967dbe8a5
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/393174
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
The List method previously listed only security advisories that did
not correspond to CVEs. Add an argument that allows the caller to
choose this behavior or the opposite, to list the advisories that do
have CVEs.
Change-Id: I286d4671d7bebe729eeef224acf8ce3fda9c21df
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/388674
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Add code to update the store state with information from GitHub's API
about its security advisories.
Since we can process the entire set of advisories in one transaction
(there are only 34 non-CVE Go-related advisories as of this writing),
this update process is considerably simpler than that for CVEs. We
don't need to worry about concurrent updates, for one thing. So we
avoid a separate record for each run, as we have with the
CommitUpdateRecord for CVEs.
Change-Id: I913864b4c83e39144af84da9447b33870c8cc5f0
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/384214
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
The Firestore client now supports service account impersonation.
This is needed to run tests against our Firestore database.
Change-Id: I7803b4713564a8320361ae88bc76ff6189463a76
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/384094
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Add support for OpenTelemetry tracing of certain
important functions, using the event package.
We don't have a burning need for traces, but they
are nice to have, and this was an important exercise
for validating the approach of the event package.
Change-Id: I37d1f56f06f425f3b1eb885877a0d2f5ac85a098
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/380440
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
Set the DOCKER_IMAGE environment variable in the container to
be the image tag.
Change-Id: Iec4c9214d9a2ae296d8c2271a729135863f2ca93
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/376296
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
Create a package for working with github.com/CVEProject/cvelist.
Change-Id: Iadac53c0f13ba658bc20a9bd4f74d58ade5a4f44
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/375716
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
Move the issue client code from internal/worker to its own package, internal/issues.
Move the ParseGitHubRepo function to the internal package.
Change-Id: I68cdb69a3273f541620a5c96f90fc66c0628e8fc
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/375715
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
Previously, the worker used structured logging almost completely:
anything that would have been a format string parameter was made a
label instead.
This led to a poor experience with the GCP logging UI. You could see
the message of each log entry at a glance, but it took two clicks to
look at the labels. For example, when the worker created a GitHub
issue for a CVE, you'd see "created issue", but you wouldn't know the
CVE or number of the issue unless you opened the log entry.
This CL rethinks the log "frontend" -- the API that programmers
actually use to write log messages -- to allow a mix of labels
and printf-style formatting. The previous approach made it easy
to do one or the other, but not to mix both in a single log entry.
You could either write
log.Info(message, labels...)
or
log.Infof(format, args...)
but not both, since you can't have two "..." parameters in a function.
In this CL's design, one can write
log.With(namesAndValues...).Infof(format, args...)
To further improve the ergonomics, the With function takes key-value
pairs rather than labels. Thus
event.Int64("limit", int64(limit))
becomes
log.With("limit", limit)
The performance is significantly worse, but for this server it
doesn't matter.
The CL also revisits each log message and makes a choice about whether
to display a value in the message, or to make a label out of it, or
both. As I said, putting the value in the message lets you see it at a
glance. The advantage of a label is you can easily query on it.
Most values only make sense as one or the other. For example, we'd
like to see the number of CVEs added during an update, but it doesn't
seem interesting to query the logs on that field. So that was put into
the message.
Change-Id: Iba2c70a14ea951c4d74335da6ac4e8265d652ded
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/373894
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
golang.org/x/vuln/srv is moved to this repository. Originally, the
motivation for creating x/vuln was to split the YAML reports and Go code
into two separate repositories. However, this resulted in a few
issues:
1. The structure of the YAML reports is tightly coupled with the structs
in internal/report, and changing one without the other would result
in errors when linting the reports.
2. The vlint package itself needed to be exported, even though the only
consumer was the test in x/vulndb.
3. The deploy/build.yaml script depends on cmd/gendb@latest, so updating
that command could easily break the script (for example, submitting
CL 373004 without changing the reference in deploy/build.yaml).
Additionally, the original location of this code was x/vuln, which
contained two types of packages.
(1) Packages meant for consumption by other clients (for example,
x/vuln/client), and
(2) Internal packages that were only meant for use to spin up the
worker.
The internal packages resulted in many dependencies since they pulled in
GCP, which we don't want clients of the vulncheck library to
have to pull in. This problem was originally solved by creating a nested
module inside x/vuln, but nicer separation that would also solve the
issues above is the following:
* x/vuln: contains Go code meant to be imported by others
* x/vulndb: contains internal code only used to maintain the vulndb
For golang/go#50247
Change-Id: I74a7b7f9b8fc5b0ad48a45fc3156f93c08aa9955
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/373495
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Tatiana Bradley
Jonathan Amsterdam
Julie Qiu
Maceo Thompson
cui fliter