Add new excluded reason, WITHDRAWN, which indicates
that a report was withdrawn before we got a chance
to publish it in vulndb.
This allows us to keep better track of withdrawn reports
(as opposed to completely omitting them from our
records).
Change-Id: I7209edc88e903787b0c79556177af8f34fed8a4e
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/607818
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Tatiana Bradley <tatianabradley@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
The post-submit trigger that deploys vulndb now
additionally publishes or updates any CVEs that
have changed via the CVE Services API.
This means we no longer need to manually publish
CVEs in the normal course of business.
This is safe because the CVE program no longer
makes direct edits to CVE records, so the records
in data/cve/v5 are canonical.
This CL additionally removes the logic to handle
the case in which a record was modified by the CVE
program, and adds a convenience command "publish-all"
which non-interactively publishes/updates all
CVEs that need it.
API user name and token for the service account
are stored in GCP Secret Manager.
Manual tests via "gcloud builds submit" worked.
Change-Id: I68ce77001067c6e1eff9478234ec7fc76dac587d
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/606779
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Use the authenticated github client instead of a direct HTTP request
to the Github API when fetching the published time for a GHSA.
This allows us to perform batch commands without being rate-limited.
Change-Id: Ie4f357ab9ec105389f6990964a86f27b77079271
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/606357
Auto-Submit: Tatiana Bradley <tatianabradley@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
When converting from GHSA OSV to YAML, preserve the withdrawn status,
and consider it in the "create-like" commands of vulnreport:
- for create, create-excluded, and unexclude, error if a report is
withdrawn (there is no need to publish a new withdrawn report)
- for regen, allow a withdrawn report (this allows us to withdraw
published UNREVIEWED reports that were later withdrawn by the source)
Change-Id: Ifafd543c7620418280d6312cb7fedf558e46d04f
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/606356
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Tatiana Bradley <tatianabradley@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Remove the "possible duplicate" label and instead
label all suspected duplicates as "duplicate" and
post a comment of the form "Duplicate of #NNN" to
the issue.
Update the instructions for the triager.
This is OK because the duplicate-finding check is
almost always correct.
Change-Id: I9d036f3a0490564000a13d783353608cde39880a
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/606236
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Add a command, whose usage is "triage CVE-XXXX-YYYY"
(or "triage GHSA-xxxx-yyyy-zzzz", but that is trivial as GHSA's
specify their ecosystem explicitly) that gives direct
access to the worker triage algorithm.
This helps with experimentation and testing of tweaks to
the algorithm. (The goal is to make this much faster
by greatly reducing the number of requests made to pkgsite).
Change-Id: I74d54e60afbb1fe7ebf26fce4ae2d079ecb63b4b
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/601379
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Separate triage-related code to its own internal
package so it can be re-used throughout.
Change-Id: I1c143624d718b896edb64afa020875925210b094
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/601378
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Fix small issues like unused symbols, unchecked errors etc.
Bump dependency on protobuf, which has a vulnerability.
Change-Id: I10385ff41302d1446c35af43ae72219fc9687150
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/601376
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Fix a bug in which the "likely duplicate" label was applied
to all issues that have duplicates on the tracker. (For example,
if #1 and #2 both refer to GHSA-xxxx-yyyy-zzzz, only one of
these should be marked as a duplicate).
This also revealed some bugs in the fake in-memory implementation
of the GHSA API, which are now fixed.
Change-Id: Ifd98befdf3e23f1fc95df38533107de9c921b195
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/599456
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Add a simple check for package existence in vulnreport fix, which
pings pkg.go.dev to determine if a package exists.
This is more likely to succeed (and faster) than the package/symbol
check which downloads the whole package. We now skip this symbol-check
when there are no symbols listed.
There are still some cases in which this fails incorrectly (e.g. if pkgsite
for some reason couldn't cache the given package/version), so the check
can be bypassed.
Change-Id: I922eae0dec9a376210f0f0fd1d70a67da934ffaa
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/599180
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Before writing YAML/OSV/CVE files, check if their contents would
be modified. The main benefit of this is that we now only print
out the written filename if its contents change, which is helpful
for determining the impact of batch operations.
Change-Id: Ieae133a1697d98b99cb40dda56826a5c46f40487
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/599179
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Unify the display of xrefs in the worker and in vulnreport xref.
Call out duplicate aliases more prominently, as they indicate a problem,
whereas module xrefs are informational.
Change-Id: I3898ab1709bb3bfd6aefcfa4aef236af5f270fa7
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/599176
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
The algorithm that determines priority for a report
relies on the affected modules. Sometimes not all affected
modules are known at the outset (e.g., because they are
fixed during report creation).
Ensure that we don't accidentally create UNREVIEWED reports
which are high priority by re-checking the priority of a report
after creating it. As an extra safeguard, also do this check in
the TestLintReports function which acts as a presubmit check.
This involves some refactoring of the priority algorithm. The only
change to the fundamental behavior is that an override list
now exists, where we can add modules that should always have a
certain priority regardless of what the priority algorithm would
say.
Also, the xref command now addionally prints out the priority decision
for a report.
Change-Id: Ia3301022678d7392fb3deb059f9a248dcb153ecc
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/598415
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Adds a command that makes it easier to withdraw an
existing report.
Usage:
$ vulnreport -reason="..." withdraw NNN
Change-Id: Iabe6c1a4b0d0ce15692bb6be743876a790dec437
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/595996
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
This was a backup option in case osv.dev did not have the GHSA yet;
this is no longer needed now that we're pulling directly from Github.
Change-Id: Ib5a1b9752eac1efe2a91ef0403771d5575180402
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/597755
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Fetch GHSA OSV from github.com/github/advisory-database instead
of osv.dev, as osv.dev sometimes makes edits to the OSV or has
an older version of it.
Unfortunately this requires making two HTTP requests: the first to
determine the published year/month of the GHSA from api.github.com,
and the second to pull the OSV from the GHSA database git repo. There
is no way (that I am aware of) to make a direct API call to get GHSAs
in OSV format.
Change-Id: I8bfd580b1e8ee38f9bc6b8afb08415e0de1a3040
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/597735
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
In the inspect command, display stats on the number of
withdrawn and unreviewed reports in the corpus.
Change-Id: I724a4f2bc00dbe279c2b20ecd9da5fcd961c029c
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/596181
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
For first-party reports where we have assigned our own CVE,
auto-populate "cve_metadata" instead of "cves".
Change-Id: Ifa23ece087f03a294e07ba4fba4267a0de890431
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/596179
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Move removal of description and packages (for UNREVIEWED reports)
to report.New so that these actions can be tested more easily.
Change-Id: Ie533f3ef5642f0866c91c28010482eec1d844739
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/595275
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Instead of storing version ranges as structs with paired
Introduced/Fixed versions, follow the OSV convention of
considering each version as its own object with an associated
type.
This simplifies operations such as sorting and merging version
lists, making it easier for us to improve automation.
The only effect on the user (vulndb maintainer) is that the
YAML syntax is now:
- introduced: xxx
- fixed: xxx
instead of
- introduced: xxx
fixed: xxx
As a convenience, however, the old format is still accepted for
writing reports. (However, it will be automatically converted to the
new format when vulnreport fix is run).
A follow up CL will make this change for all existing YAML reports.
This will NOT affect the published OSV files.
Change-Id: I91c524b311be5230db5d382f77de4a8e0cd1dda7
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/593820
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
There is an unknown bug causing unmarshal for OSV Severity field
to not work.
We don't use this field, so to work around this issue for now,
simply ignore the Severity field.
To do this, the dependency on osv-scanner was removed and the
relevant files were copied and modified as needed.
Change-Id: I956ea5d2c9c19f2992e6a1c9b723cea35f5e92d6
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/593817
Auto-Submit: Tatiana Bradley <tatianabradley@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Record the reason a report was previously excluded when
unexcluding it. This will allow us to take this info into account
when deciding the priority of new reports.
Change-Id: I6ad08f28ca7f9bec78280f30db35b0b6546085db
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/592776
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
1) For commit, allow the user to specify a batch size with -batch=X
2) For fix, allow the user to skip all non-lint checks with -skip-checks
Change-Id: I1a42e793cecae6f3086c613e63f410e664f4cce8
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/592758
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
When using the existing corpus to prioritize new vulnerabilities,
treat unreviewed unexcluded reports the same as likely-binary excluded
reports.
Change-Id: Id803d05ecd33b4486086acac8ff124977b3725ef
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/592777
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Previously, vulnreport triage only considered a subset of open
issues (namely, the ones being processed by the command) when
looking for duplicates. This meant that the command only worked
properly when operating on all open issues.
The command now considers all open issues when looking for duplicates.
Change-Id: Iefe17a46503e50bccdd7dc43561999aa1fae4db0
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/592757
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Command vulnreport review converts an unreviewed report
to a reviewed one, regenerating the report from the latest
version of the source and leaving TODOs as appropriate.
Change-Id: Ifc2bf85b00e5495852af6bd5086b6dc402cbebb2
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/592775
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Adds a command that allows the user (authenticated as a CNA) to
"reject" a CVE that will never be published.
This is intended for CVEs that were reserved but will never be assigned
to a vulnerability because, e.g, the year is not current.
Change-Id: Id60a0e5417d43e791ada898ff83bcef2563c2322
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/592435
Auto-Submit: Tatiana Bradley <tatianabradley@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Issues labeled OUT_OF_SCOPE should not get a report at all,
so skip them in "vulnreport create".
Change-Id: Ic7051c1ca96e1836653f4f5fc5633a771ccec805
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/592455
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Adds a test framework that allows tests to inject fake/mock
dependencies into the vulnreport commands and record the expected
output of commands.
Some subcommand tests are left as TODOs, as they require additional
fake/mock dependencies that haven't been implemented yet.
Change-Id: I25f6085f2297e5b9d916f0927c1111ac2b49bef8
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/590038
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Where possible, unify Firestore operations so that there
are not separate functions for adding/updating/fetching
a record based on its type (CVE/GHSA). The intended operation
is inferred by given ID(s).
Change-Id: Ic82e3ab4c9d519c3101f95444bc0ad306fa2a14e
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/588759
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Rename CVERecord->CVE4Record and GHSARecord->LegacyGHSARecord.
Both of these need to be updated to support CVE JSON 5 and
GHSA OSV record formats, respectively, so this change makes
it more clear where the old formats are being relied on.
Change-Id: Ib339f0addbc16c37ed03383d64e7cdb30165f366
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/588758
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Refactor worker code so that the body of issues created from
CVEs and GHSAs have the same basic structure.
Change-Id: Icf082de5642fbb2c13bbb0478916afed52548585
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/586139
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
When creating unreviewed reports, automatically remove references
that do not exist.
Also remove package-level data, as it can cause false-negatives
if it is not correct.
(For reviewed reports, we preserve these pieces of info as a human
will review them and manually determine if they are useful).
Change-Id: I2ff6bde62320d2f56f9d5a67ef438f4cafbaf6e5
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/591200
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Commit-Queue: Tatiana Bradley <tatianabradley@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Modify the prioritization algorithm so that it only considers the number
of reviewed vs. likely-binary reports, instead of comparing the number
of excluded vs. non-excluded reports. This means that the number of unreviewed
reports does not (currently) affect the prioritization result.
In addition, separate the code used to prioritize modules (for vulnreport triage)
into its own package so it can be tested in isolation.
Also add a basic command line tool "priority" that can be used to find the
priority result for a module directly.
Change-Id: Ic7ebe76d8f5091f56bc3eb65a5064391136b2064
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/591195
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Commit-Queue: Tatiana Bradley <tatianabradley@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Command vulnreport regen regenerates a report based on
the latest version of the source.
Intended for UNREVIEWED reports.
Use the following command to regenerate all unreviewed
reports:
$ vulnreport regen data/reports/*.yaml
Change-Id: I3f956fde473b8375bd523049118d8f6817aad9ae
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/590856
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Tatiana Bradley
Ian Cottrell