This adds a new builtin, `git update-microsoft-git`, that executes the platform-specific upgrade steps to get the latest version of `microsoft-git`.
On Windows, this means running `git update-git-for-windows` which was updated to use the `microsoft/git` releases page, when appropriate. See #321 for details.
On macOS, this means running a sequence of `brew` commands. These are adapted from the `UpgradeVerb` in `microsoft/scalar`, with an important simplification: we don't need to differentiate between the `scalar` and `scalar-azrepos` cask.
When the virtualfilesystem is enabled the previous implementation of
clear_ce_flags would iterate all of the cache entries and query whether
each one is in the virtual filesystem to determine whether to clear one
of the SKIP_WORKTREE bits. For each cache entry, we would do a hash
lookup for each parent directory in the is_included_in_virtualfilesystem
function.
The former approach is slow for a typical Windows OS enlistment with
3 million files where only a small percentage is in the virtual
filesystem. The cost is
O(n_index_entries * n_chars_per_path * n_parent_directories_per_path).
In this change, we use the same approach as apply_virtualfilesystem,
which iterates the set of entries in the virtualfilesystem and searches
in the cache for the corresponding entries in order to clear their
flags. This approach has a cost of
O(n_virtual_filesystem_entries * n_chars_per_path * log(n_index_entries)).
The apply_virtualfilesystem code was refactored a bit and modified to
clear flags for all names that 'alias' a given virtual filesystem name
when ignore_case is set.
n_virtual_filesystem_entries is typically much less than
n_index_entries, in which case the new approach is much faster. We wind
up building the name hash for the index, but this occurs quickly thanks
to the multi-threading.
This PR updates our `vfs-2.29.0` branch's version of `git maintenance` to match the latest in upstream. Unfortunately, not all of these commits made it to the `2.30` release candidate, but there are more commits from the series making it in. They will cause a conflict in the `vfs-2.30.0` rebase, so merge them in here. This also includes the `fixup!` reverts of the earlier versions.
Finally, I also noticed that we started depending on `git maintenance start` in Scalar for macOS, but we never checked that this worked with the shared object cache. It doesn't! 😨 The very tip commit of this PR includes logic to make `git maintenance run` care about `gvfs.sharedCache`. Functional test updates in Scalar will follow.
Add virtual file system settings and hook proc. On index load,
clear/set the skip worktree bits based on the virtual file system data.
Use virtual file system data to update skip-worktree bit in
unpack-trees. Use virtual file system data to exclude files and folders
not explicitly requested.
The hook was first contributed in private, but was extended via the
following pull requests:
#15#27#33#70
Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Most of these were done in private before microsoft/git. However,
the following pull requests modified the core feature:
#85#89#91#98#243#263
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
These are two highly-requested items from an internal team considering a
move to Scalar using Azure Repos.
1. Remove the requirement that we create a `src` directory at clone time.
2. Allow `git worktree` even when using the GVFS protocol.
These are not difficult to implement. The `--no-src` option could even
be submitted upstream (though the commit will need to drop one bit about
an interaction with the local cache path).
When running 'scalar reconfigure -a', such as at install time, Scalar
has warning messages about the repository missing (or not containing a
.git directory). Failures can also happen while trying to modify the
repository-local config for that repository.
These warnings may seem confusing to users who don't understand what
they mean or how to stop them.
Add a warning that instructs the user how to remove the warning in
future installations.
Some users have strong aversions to Scalar's opinion that the repository
should be in a 'src' directory, even though it creates a clean slate for
placing build outputs in adjacent directories.
The --no-src option allows users to opt-out of the default behavior. The
parse-opt logic automatically figures out that '--src' is a possible
flag that negates '--no-src'.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
This allows fixing settings after a Scalar upgrade, or after botching
the enlistments configuration.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When running 'scalar reconfigure -a', such as at install time, Scalar
has warning messages about the repository missing (or not containing a
.git directory). Failures can also happen while trying to modify the
repository-local config for that repository.
These warnings may seem confusing to users who don't understand what
they mean or how to stop them.
Add a warning that instructs the user how to remove the warning in
future installations.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
We should not be putting the .scalarCache inside the enlistment as a
sibling to the 'src' directory. This only happens in "unattended" mode,
but it also negates any benefit of a shared object cache because each
enlistment absolutely does not share any objects with others.
Move the shared object cache in this case to a level above the
enlistment, so at least there is some hope that it can be reused. This
is also critical to the upcoming --no-src option, since the shared
object cache cannot be located within the Git repository.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
When FSMonitor was upstreamed, the 'core.useBuiltinFSMonitor' config was
deprecated and replaced with an overload of the 'core.fsmonitor' config
(i.e., if a boolean value was specified in 'core.fsmonitor', it is treated
the way 'core.useBuiltinFSMonitor' originally was). Because 'scalar
register' actively sets that config, use it to upgrade the deprecated config
setting.
Co-authored-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Victoria Dye <vdye@github.com>
There are many reasons why discovering a Git directory may fail. In
particular, 8959555cee (setup_git_directory(): add an owner check for
the top-level directory, 2022-03-02) added ownership checks as a
security precaution.
Callers attempting to set up a Git directory may want to inform the user
about the reason for the failure. For that, expose the enum
discovery_result from within setup.c and into cache.h where
discover_git_directory() is defined.
I initially wanted to change the return type of discover_git_directory()
to be this enum, but several callers rely upon the "zero means success".
The two problems with this are:
1. The zero value of the enum is actually GIT_DIR_NONE, so nonpositive
results are errors.
2. There are multiple successful states, so some positive results are
successful.
Instead of updating all callers immediately, add a new method,
discover_git_directory_reason(), and convert discover_git_directory() to
be a thin shim on top of it.
Because there are extra checks that discover_git_directory_reason() does
after setup_git_directory_gently_1(), there are other modes that can be
returned for failure states. Add these modes to the enum, but be sure to
explicitly add them as BUG() states in the switch of
setup_git_directory_gently().
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
The strip_last_component() method is helpful for finding the parent
directory of a path stored in a strbuf. Extract it to a global method
advertised in abspath.h. With that additional visibility, it is helpful to
rename it to be more specific to paths.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
Prepare `scalar` to use the GVFS protocol instead of partial clone
(required to support Azure Repos).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Scalar's Functional Test suite is pretty comprehensive, and caught more
than just one bug in the built-in FSMonitor that was missed by Git's own
test suite.
To benefit from this test suite, automatically run it on the `vfs-*`
and `features/*` branches.
Note: for simplicity, we're building Git from scratch in all matrix
jobs.
Also note: for speed, we are using `git-sdk-64-minimal`, even if it
lacks the `/bin/install` that we need to install Git's files; We're
providing a minimal shell script shim instead. Also, we do not need to
bother with the Tcl/Tk parts, therefore we're skipping them, too.
Finally, we use GIT_FORCE_UNTRACKED_CACHE in the functional tests, to
give the untracked cache a thorough work-out.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Those tests specifically verify that the `.zip` file path is shown on
`stdout`. Let's do that again, under the assumption that there are
scripts out there that rely on this behavior.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Scalar in Microsoft's Git fork can do a little more than Scalar in
upstream Git: in Microsoft's Git, it supports the GVFS protocol so that
Scalar can clone from Azure DevOps.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This merges the upstreamable part of the Scalar patches.
Minor merge conflicts (caused by the gvfs-helper) were resolved
trivially.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In Scalar's functional tests, we do not do anything with authentication.
Therefore, we do want to avoid accessing the `vsts/info` endpoint
because it requires authentication even on otherwise public
repositories.
Let's introduce the environment variable `SCALAR_TEST_SKIP_VSTS_INFO`
which can be set to `true` to simply skip that step (and force the
`url_*` style repository IDs instead of `id_*` whenever possible).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The .NET version supported running `scalar config` to reconfigure the
current enlistment, and now the C port does, too.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This allows setting the GVFS-enabled cache server, or listing the one(s)
associated with the remote repository.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Use a fixed 3 tries total to see how that increases our chances of
success for subcommands such as 'git fetch'.
We special-case the `diagnose` command here: When 672196a307
(scalar-diagnose: use 'git diagnose --mode=all', 2022-08-12) updated
'scalar diagnose' to run 'git diagnose' as a subprocess, it was passed
through the run_git() caller. We need to avoid repeating the call when
the underlying 'git diagnose' command fails.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Azure Repos does not support partial clones at the moment, but it does
support the GVFS protocol. To that end, the Microsoft fork of Git has a
`gvfs-helper` command that is optionally used to perform essentially the
same functionality as partial clone.
Let's verify that `scalar clone` detects that situation and enables the
GVFS helper.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This finalizes the port of the `QueryVstsInfo()` function: we already
taught `gvfs-helper` to access the `vsts/info` endpoint on demand, we
implemented proper JSON parsing, and now it is time to hook it all up.
To that end, we also provide a default local cache root directory. It
works the same way as the .NET version of Scalar: it uses
C:\scalarCache on Windows,
~/.scalarCache/ on macOS and
~/.cache/scalar on Linux
Modified to include call to is_unattended() that was removed from a
previous commit.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
On Windows, both the forward slash and the backslash are directory
separators. Which means that `a\b\c` really is inside `a/b`. Therefore,
we need to special-case the directory separators in the helper function
`cmp_icase()` that is used in the loop in `dir_inside_of()`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We already have the `config` command that accesses the `gvfs/config`
endpoint.
To implement `scalar`, we also need to be able to access the `vsts/info`
endpoint. Let's add a command to do precisely that.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This comes in handy, as we want to verify that `scalar clone` also works
against a GVFS-enabled remote repository.
Note that we have to set `MSYS2_ENV_CONV_EXCL` to prevent MSYS2 from
mangling `PATH_TRANSLATED`: The value _does_ look like a Unix-style
path, but no, MSYS2 must not be allowed to convert that into a Windows
path: `http-backend` needs it in the unmodified form. (The MSYS2 runtime
comes in when `git` is run via `bin-wrappers/git`, which is a shell
script.)
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
With this change, we come a big step closer to feature parity with
Scalar: this allows cloning from Azure Repos (which do not support
partial clones at time of writing).
We use the just-implemented JSON parser to parse the response we got
from the `gvfs/config` endpoint; Please note that this response might,
or might not, contain information about a cache server. The presence or
absence of said cache server, however, has nothing to do with the
ability to speak the GVFS protocol (but the presence of the
`gvfs/config` endpoint does that).
An alternative considered during the development of this patch was to
perform simple string matching instead of parsing the JSON-formatted
data; However, this would have been fragile, as the response contains
free-form text (e.g. the repository's description) which might contain
parts that would confuse a simple string matcher (but not a proper JSON
parser).
Note: we need to limit the re-try logic in `git clone` to handle only
the non-GVFS case: the call to `set_config()` to un-set the partial
clone settings would otherwise fail because those settings would not
exist in the GVFS protocol case. This will at least give us a clearer
reason why such a fetch fails.
Co-authored-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Johannes Schindelin
Derrick Stolee
Victoria Dye
Victoria Dye
Johannes Schindelin
Jeff Hostetler
Jeff Hostetler
Johannes Schindelin
Derrick Stolee