This commit adds telemetry capturing for command execution. The data
captured explicitly captured and sent to application insights is only
the command id, execution time, and command completion status. We also
capture errors thrown by any command execution, but these are not sent
to application insights.
Telemetry capturing is opt-in. No data will be sent to application
insights unless the user explicitly allows it.
There are two new config settings added. The first controls whether or
not telemetry should be sent. This setting AND the global telemetry setting
must be enabled in order for telemetry to be sent.
The second setting controls whether or not telemetry event data should
be logged to the extension console. The hope here is that users can
inspect exactly what data is sent to the server and can have confidence
that nothing concerning is being leaked.
Note that the global setting for disabling telemetry collection is
handled inside the `vscode-extension-telemetry` package implicitly, so
this extension doesn't touch that setting explicitly.
The `codeql.canary` setting is being used to add an additional flag to
telemetry events. This flag will help us determine if a user in internal
or not.
The application insights key is injected at build time through a
repository secret.
This commit also includes a new `TELEMETRY.md` file that explains what
is being captured, and why.
This commit fixes#733. It does it by ensuring that the query server
emits an event when it restarts the query server. The database manager
listens for this even and properly re-registers its databases.
A few caveats though:
1. Convert query restarts to using a command that includes progress.
This will ensure that errors on restart are logged properly.
2. Because we want to log errors, we cannot use the vscode standard
EventEmitters. They run in the next tick and therefore any errors
will not be associated with this command execution.
3. Update the default cli version to run integration tests against to
2.4.2.
4. Add a new integration test that fails if databases are not
re-registered.
Also, hide the `CodeQL: Run Queries` command inside of zip folders
since we do not allow queries to be in archives. I wish we could be more
specific about when to show that command, eg- only *.ql files and
directories, but I couldn't find a way to restrict a command to only
appear on directories.
This removes the cached treeItem that is a property of the
completedQuery. We should not be caching them since they are cached by
the vscode api itself. Instead, we should recreate whenever requested.
Also, this change fixes#598 in a new way. Instead of adding the
context to the cached treeItem, we simply refresh only the item that has
changed. This is a fast operation.
This change avoids popping up error messages in two cases:
1. When doing test discovery, do not run discovery on non-existant
directories. Also, if there is an error, print to the log, and do not
pop up an error window. The reason is that test discovery is a
background operation and these should not normally cause pop-ups.
2. When looking for orphaned databases, don't pop up an error if the
storagePath can't be found. This is normal when working in a new,
single root workspace.
This fixes two classes of DBs that can't be installed directly from
downloading:
1. DBs whose central directories do not align with their file headers.
We need to download and save the entire archive before we can read
the central directory and use that to guide the unzipping.
2. Large DBs require too much memory so can't be downloaded and unzipped
in a single stream.
We also add proper progress notifications to the download progress
monitor so users are aware of how many more MBs are left to download.
It's not yet possible to do the same for unzipping using the current
unzipper library, since unzipping using the central directory does not
expose a stream.
Co-authored-by: Alexander Eyers-Taylor <alexet@github.com>
1. Clicking on query history menu items when nothing is selected. Error
message is clearer. It would be better to disable when nothing is
selected, but waiting on
https://github.com/microsoft/vscode/issues/99767 to be released.
2. Trying to run query with a missing or invalid qlpack has better
message.
3. Better hover text for "Open query".
Co-authored-by: Aditya Sharad <6874315+adityasharad@users.noreply.github.com>
This change will only work on databases created by cli >= 2.4.1. In that
version, a new `primaryLanguage` field in the `codeql-database.yml`
file. We use this property as the language.
This change also includes a refactoring of the logic around extracting
database information heuristically based on file location. As much
as possible, it is extracted to the `helpers` module. Also, the
initial quick query text is generated based on the language (if known)
otherwise it falls back to the old style of generation.
* Fix dubious index check
* Add unit tests for add/remove database
In order to do this, needed to move `databases.test.ts` to the
`minimal-workspace` test folder because these tests require that there
be some kind of workspace open in order to check on workspace folders.
Unfortunately, during tests vscode does not allow you to convert from a
single root workspace to multi-root and so several of the workspace
functions needed to be stubbed out.
* Update changelog
Co-authored-by: Andrew Eisenberg <aeisenberg@github.com>
Cleans orphan databases on startup. This commit also bumps the fs-extra
dependency to get readdir with dirent objects.
Adds the `asyncFilter` to filter arrays asynchronously.
The diff and the errors were always available, but they were not being
sent to the output.
Additionally, make sure to send output to both the test explorer log and
the codeql test log.
This fixes a bug where if there are special characters in a database
path, it is not possible to navigate to that file from the results view.
Note that the results from our BQRS returned properly encoded URIs, but
our paths coming from sarif were unencoded. Our path parsing handled
the latter correctly (even though these are not correct URIs) and the
former incorrectly.
The fix here is to first ensure all uris are properly encoded. We do
this by running `encodeURI` in sarif-utils (can't run encodeURIComponent
or else the path separators `/` will also be encoded).
Then, we ensure that when we resolve locations, we decode all file
paths.
This works in all cases I have tried. I still have an issue with running
View AST on some of these databases, but that I believe is a separate
issue.
Split commandRunner into two functions: commandRunner and
commandRunnerWithProgress.
Also, take advantage of default arguments for ProgressOptions.
And updates changelog.
When a user runs multiple queries on a non-upgraded database, ensure
that only one dialog appears for upgrade.
This commit also migrates the upgrades.ts file to using the passed-in
cancellation token and progress monitor. This ensures that cancelling
a database upgrade command will also cancel out of any wrapper
operations.
Fixes#534
Andrew Eisenberg
aeisenberg
Max Schaefer
github-actions[bot]