When FSMonitor was upstreamed, the 'core.useBuiltinFSMonitor' config was
deprecated and replaced with an overload of the 'core.fsmonitor' config
(i.e., if a boolean value was specified in 'core.fsmonitor', it is treated
the way 'core.useBuiltinFSMonitor' originally was). Because 'scalar
register' actively sets that config, use it to upgrade the deprecated config
setting.
Co-authored-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Victoria Dye <vdye@github.com>
There are many reasons why discovering a Git directory may fail. In
particular, 8959555cee (setup_git_directory(): add an owner check for
the top-level directory, 2022-03-02) added ownership checks as a
security precaution.
Callers attempting to set up a Git directory may want to inform the user
about the reason for the failure. For that, expose the enum
discovery_result from within setup.c and into cache.h where
discover_git_directory() is defined.
I initially wanted to change the return type of discover_git_directory()
to be this enum, but several callers rely upon the "zero means success".
The two problems with this are:
1. The zero value of the enum is actually GIT_DIR_NONE, so nonpositive
results are errors.
2. There are multiple successful states, so some positive results are
successful.
Instead of updating all callers immediately, add a new method,
discover_git_directory_reason(), and convert discover_git_directory() to
be a thin shim on top of it.
Because there are extra checks that discover_git_directory_reason() does
after setup_git_directory_gently_1(), there are other modes that can be
returned for failure states. Add these modes to the enum, but be sure to
explicitly add them as BUG() states in the switch of
setup_git_directory_gently().
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
The strip_last_component() method is helpful for finding the parent
directory of a path stored in a strbuf. Extract it to a global method
advertised in abspath.h. With that additional visibility, it is helpful to
rename it to be more specific to paths.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
Prepare `scalar` to use the GVFS protocol instead of partial clone
(required to support Azure Repos).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Scalar's Functional Test suite is pretty comprehensive, and caught more
than just one bug in the built-in FSMonitor that was missed by Git's own
test suite.
To benefit from this test suite, automatically run it on the `vfs-*`
and `features/*` branches.
Note: for simplicity, we're building Git from scratch in all matrix
jobs.
Also note: for speed, we are using `git-sdk-64-minimal`, even if it
lacks the `/bin/install` that we need to install Git's files; We're
providing a minimal shell script shim instead. Also, we do not need to
bother with the Tcl/Tk parts, therefore we're skipping them, too.
Finally, we use GIT_FORCE_UNTRACKED_CACHE in the functional tests, to
give the untracked cache a thorough work-out.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Those tests specifically verify that the `.zip` file path is shown on
`stdout`. Let's do that again, under the assumption that there are
scripts out there that rely on this behavior.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Scalar in Microsoft's Git fork can do a little more than Scalar in
upstream Git: in Microsoft's Git, it supports the GVFS protocol so that
Scalar can clone from Azure DevOps.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This merges the upstreamable part of the Scalar patches.
Minor merge conflicts (caused by the gvfs-helper) were resolved
trivially.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In Scalar's functional tests, we do not do anything with authentication.
Therefore, we do want to avoid accessing the `vsts/info` endpoint
because it requires authentication even on otherwise public
repositories.
Let's introduce the environment variable `SCALAR_TEST_SKIP_VSTS_INFO`
which can be set to `true` to simply skip that step (and force the
`url_*` style repository IDs instead of `id_*` whenever possible).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The .NET version supported running `scalar config` to reconfigure the
current enlistment, and now the C port does, too.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This allows setting the GVFS-enabled cache server, or listing the one(s)
associated with the remote repository.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Use a fixed 3 tries total to see how that increases our chances of
success for subcommands such as 'git fetch'.
We special-case the `diagnose` command here: When 672196a307
(scalar-diagnose: use 'git diagnose --mode=all', 2022-08-12) updated
'scalar diagnose' to run 'git diagnose' as a subprocess, it was passed
through the run_git() caller. We need to avoid repeating the call when
the underlying 'git diagnose' command fails.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Azure Repos does not support partial clones at the moment, but it does
support the GVFS protocol. To that end, the Microsoft fork of Git has a
`gvfs-helper` command that is optionally used to perform essentially the
same functionality as partial clone.
Let's verify that `scalar clone` detects that situation and enables the
GVFS helper.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This finalizes the port of the `QueryVstsInfo()` function: we already
taught `gvfs-helper` to access the `vsts/info` endpoint on demand, we
implemented proper JSON parsing, and now it is time to hook it all up.
To that end, we also provide a default local cache root directory. It
works the same way as the .NET version of Scalar: it uses
C:\scalarCache on Windows,
~/.scalarCache/ on macOS and
~/.cache/scalar on Linux
Modified to include call to is_unattended() that was removed from a
previous commit.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
On Windows, both the forward slash and the backslash are directory
separators. Which means that `a\b\c` really is inside `a/b`. Therefore,
we need to special-case the directory separators in the helper function
`cmp_icase()` that is used in the loop in `dir_inside_of()`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We already have the `config` command that accesses the `gvfs/config`
endpoint.
To implement `scalar`, we also need to be able to access the `vsts/info`
endpoint. Let's add a command to do precisely that.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This comes in handy, as we want to verify that `scalar clone` also works
against a GVFS-enabled remote repository.
Note that we have to set `MSYS2_ENV_CONV_EXCL` to prevent MSYS2 from
mangling `PATH_TRANSLATED`: The value _does_ look like a Unix-style
path, but no, MSYS2 must not be allowed to convert that into a Windows
path: `http-backend` needs it in the unmodified form. (The MSYS2 runtime
comes in when `git` is run via `bin-wrappers/git`, which is a shell
script.)
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
With this change, we come a big step closer to feature parity with
Scalar: this allows cloning from Azure Repos (which do not support
partial clones at time of writing).
We use the just-implemented JSON parser to parse the response we got
from the `gvfs/config` endpoint; Please note that this response might,
or might not, contain information about a cache server. The presence or
absence of said cache server, however, has nothing to do with the
ability to speak the GVFS protocol (but the presence of the
`gvfs/config` endpoint does that).
An alternative considered during the development of this patch was to
perform simple string matching instead of parsing the JSON-formatted
data; However, this would have been fragile, as the response contains
free-form text (e.g. the repository's description) which might contain
parts that would confuse a simple string matcher (but not a proper JSON
parser).
Note: we need to limit the re-try logic in `git clone` to handle only
the non-GVFS case: the call to `set_config()` to un-set the partial
clone settings would otherwise fail because those settings would not
exist in the GVFS protocol case. This will at least give us a clearer
reason why such a fetch fails.
Co-authored-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
No grown-up C project comes without their own JSON parser.
Just kidding!
We need to parse a JSON result when determining which cache server to
use. It would appear that searching for needles `"CacheServers":[`,
`"Url":"` and `"GlobalDefault":true` _happens_ to work right now, it is
fragile as it depends on no whitespace padding and on the order of the
fields remaining as-is.
Let's implement a super simple JSON parser (at the cost of being
slightly inefficient) for that purpose. To avoid allocating a ton of
memory, we implement a callback-based one. And to save on complexity,
let's not even bother validating the input properly (we will just go
ahead and instead rely on Azure Repos to produce correct JSON).
Note: An alternative would have been to use existing solutions such as
JSON-C, CentiJSON or JSMN. However, they are all a lot larger than the
current solution; The smallest, JSMN, which does not even provide parsed
string values (something we actually need) weighs in with 471 lines,
while we get away with 182 + 29 lines for the C and the header file,
respectively.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Microsoft's fork of Git is not quite Git for Windows, therefore we want
to tell the keen reader all about it. :-)
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
The steps to update the microsoft-git cask are:
1. brew update
2. brew upgrade --cask microsoft-git
This is adapted from the UpgradeVerb within microsoft/scalar. There is
one important simplification: Scalar needed to check 'brew list --cask'
to find out if the 'scalar' cask or the 'scalar-azrepos' cask was
installed (which determined if the 'microsoft-git' cask was a necessary
dependency). We do not need that here, since we are already in the
microsoft-git cask.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Add basic installer validation to release pipeline for Windows, macOS, and
Linux (Debian package only). Validation runs the installers/any necessary
setup and checks that the installed version matches the expected version.
In 'git-update-git-for-windows', there is a recently_seen variable that
is loaded from Git config. This is intended to allow users to say "No, I
don't want that version of Git for Windows." If users say no, then they
are not reminded. Ever.
We want users of microsoft/git to be notified repeately until they
upgrade. The first notification might be dismissed because they don't
want to interrupt their work. They should get the picture within a few
reminders and upgrade in a timely fashion.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Allow concurrent `scalar register` and `scalar unregister` calls to be
more collaborative when trying to lock the global Git config at the very
same time.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These docs have been altered to fit the version implemented in C within
microsoft/git. This means in particular that the advanced.md file no
longer applied at all. Some other areas were removed or significantly
edited.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
We have long inherited the pull request template from
git-for-windows/git, but we should probably do a better job of
specifying the need for why a PR in microsoft/git exists instead of an
upstream contribution.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
On Windows, we have the 'git update-git-for-windows' command. It is
poorly named within the microsoft/git fork, because the script has been
updated to look at the GitHub releases of microsoft/git, not
git-for-windows/git.
Still, it handles all the complicated details about downloading,
verifying, and running the installer.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Set the `GIT_BUILT_FROM_COMMIT` based on the version specified in the `make
dist` output archive header. This ensures the commit hash is shown in
`git version --build-options`.
Signed-off-by: Victoria Dye <vdye@github.com>
By default, Git fails immediately when locking a config file for writing
fails due to an existing lock. With this change, Scalar-registered
repositories will fall back to trying a couple times within a 150ms
timeout.
Signed-off-by: Johannes Schindelin <johasc@microsoft.com>
We have been using the default issue template from git-for-windows/git,
but we should ask different questions than Git for Windows. Update the
issue template to ask these helpful questions.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
Just do the boilerplate stuff of making a new builtin, including
documentation and integration with git.c.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Update `git archive` tree-ish argument from `HEAD^{tree}` to `HEAD`. By
using a commit (rather than tree) reference, the commit hash will be stored
as an extended pax header, extractable git `git get-tar-commit-id`.
The intended use-case for this change is building `git` from the output of
`make dist` - in combination with the ability to specify a fallback
`GIT_BUILT_FROM_COMMIT`, a user can extract the commit ID used to build the
archive and set it as `GIT_BUILT_FROM_COMMIT`. The result is fully-populated
information for the commit hash in `git version --build-options`.
Signed-off-by: Victoria Dye <vdye@github.com>
- sign using Azure-stored certificates & client
- sign on Windows agent via python script
- job skipped if credentials for accessing certificate aren't present
In particular when multiple processes want to write to the config
simultaneously, it would come in handy to not fail immediately when
another process locked the config, but to gently try again.
This will help with Scalar's functional test suite which wants to
register multiple repositories for maintenance semi-simultaneously.
As not all code paths calling this function read the config (e.g. `git
config`), we have to read the config setting via
`git_config_get_ulong()`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Allow specification of a custom `GIT_BUILT_FROM_COMMIT` string to replace
the output of `git rev-parse HEAD`. This allows a build of `git` from
somewhere other than an active clone of `git` (e.g. from the archive created
with `make dist`) to include commit information in
`git version --build-options`.
Signed-off-by: Victoria Dye <vdye@github.com>
This was disabled by a0da6deeec (ci: only run win+VS build & tests in
Git for Windows' fork, 2022-12-19) to avoid other forks doing too many
builds. But we want to keep these builds for the microsoft/git fork.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
When the virtualfilesystem is enabled the previous implementation of
clear_ce_flags would iterate all of the cache entries and query whether
each one is in the virtual filesystem to determine whether to clear one
of the SKIP_WORKTREE bits. For each cache entry, we would do a hash
lookup for each parent directory in the is_included_in_virtualfilesystem
function.
The former approach is slow for a typical Windows OS enlistment with
3 million files where only a small percentage is in the virtual
filesystem. The cost is
O(n_index_entries * n_chars_per_path * n_parent_directories_per_path).
In this change, we use the same approach as apply_virtualfilesystem,
which iterates the set of entries in the virtualfilesystem and searches
in the cache for the corresponding entries in order to clear their
flags. This approach has a cost of
O(n_virtual_filesystem_entries * n_chars_per_path * log(n_index_entries)).
The apply_virtualfilesystem code was refactored a bit and modified to
clear flags for all names that 'alias' a given virtual filesystem name
when ignore_case is set.
n_virtual_filesystem_entries is typically much less than
n_index_entries, in which case the new approach is much faster. We wind
up building the name hash for the index, but this occurs quickly thanks
to the multi-threading.
Signed-off-by: Neeraj Singh <neerajsi@ntdev.microsoft.com>
For Scalar and VFS for Git, we use an alternate as a shared object
cache. We need to enable the maintenance builtin to work on that
shared object cache, especially in the background.
'scalar run <task>' would set GIT_OBJECT_DIRECTORY to handle this.
We set GIT_OBJECT_DIRECTORY based on the gvfs.sharedCache config,
but we also need the checks in pack_loose() to look at that object
directory instead of the current ODB's.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Test t5516-fetch-push.sh has a test 'deny fetch unreachable SHA1,
allowtipsha1inwant=true' that checks stderr for a specific error
string from the remote. In some build environments the error sent
over the remote connection gets mingled with the error from the
die() statement. Since both signals are being output to the same
file descriptor (but from parent and child processes), the output
we are matching with grep gets split.
To reduce the risk of this failure, follow this process instead:
1. Write an error message to stderr.
2. Write an error message across the connection.
3. exit(1).
This reorders the events so the error is written entirely before
the client receives a message from the remote, removing the race
condition.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
`sparse` complains with an error message like this:
gvfs-helper.c:2912:17: error: expression using sizeof on a
function
The culprit is this line:
curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite);
Similar lines exist in `http-push.c` and other files that are in
upstream Git, and to avoid these bogus warnings, they are already
exempted from `sparse`'s tender, loving care. We simply add
`gvfs-helper.c` to that list.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Victoria Dye
Derrick Stolee
Johannes Schindelin
Derrick Stolee
Lessley Dennington
Lessley Dennington
Johannes Schindelin
Neeraj Singh