Some users have strong aversions to Scalar's opinion that the repository
should be in a 'src' directory, even though it creates a clean slate for
placing build outputs in adjacent directories.
The --no-src option allows users to opt-out of the default behavior. The
parse-opt logic automatically figures out that '--src' is a possible
flag that negates '--no-src'.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
We should not be putting the .scalarCache inside the enlistment as a
sibling to the 'src' directory. This only happens in "unattended" mode,
but it also negates any benefit of a shared object cache because each
enlistment absolutely does not share any objects with others.
Move the shared object cache in this case to a level above the
enlistment, so at least there is some hope that it can be reused. This
is also critical to the upcoming --no-src option, since the shared
object cache cannot be located within the Git repository.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
Azure Repos does not support partial clones at the moment, but it does
support the GVFS protocol. To that end, the Microsoft fork of Git has a
`gvfs-helper` command that is optionally used to perform essentially the
same functionality as partial clone.
Let's verify that `scalar clone` detects that situation and enables the
GVFS helper.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This comes in handy, as we want to verify that `scalar clone` also works
against a GVFS-enabled remote repository.
Note that we have to set `MSYS2_ENV_CONV_EXCL` to prevent MSYS2 from
mangling `PATH_TRANSLATED`: The value _does_ look like a Unix-style
path, but no, MSYS2 must not be allowed to convert that into a Windows
path: `http-backend` needs it in the unmodified form. (The MSYS2 runtime
comes in when `git` is run via `bin-wrappers/git`, which is a shell
script.)
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The maintenance.lock file exists to prevent concurrent maintenance
processes from writing to a repository at the same time. However, it has
the downside of causing maintenance to start failing without recovery if
there is any reason why the maintenance command failed without cleaning
up the lock-file.
This change makes it such that maintenance will delete a lock file that
was modified over 6 hours ago. This will auto-heal repositories that are
stuck with failed maintenance (and maybe it will fail again, but we will
get a message other than the lock file exists).
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
* t1092: remove the 'git update-index' test that currently fails
because the command ignores the bad path, but doesn't return a
failure.
* dir.c: prevent matching against sparse-checkout patterns when the
virtual filesystem is enabled. Should prevent some corner case
issues.
* t1092: add quiet mode for some rebase tests because the stderr
output can change in some of the modes.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Test cases specific to handling untracked files in `git stash` a) ensure
that files outside the sparse checkout definition are handled as-expected
and b) document the index expansion inside of `git stash -u`. Note that, in b),
it is not the full repository index that is expanded - it is the temporary,
standalone index containing the stashed untracked files only.
Signed-off-by: Victoria Dye <vdye@github.com>
This verifies that `diff` and `diff --staged` behave the same in sparse
index repositories in the following partially-staged scenarios (i.e. the
index, HEAD, and working directory differ at a given path):
1. Path is within sparse-checkout cone.
2. Path is outside sparse-checkout cone.
3. A merge conflict exists for paths outside sparse-checkout cone.
Signed-off-by: Lessley Dennington <lessleydennington@gmail.com>
There is some strangeness when expanding a sparse-index that exists
within a submodule. We will need to resolve that later, but for now,
let's do a better job of explicitly disabling the sparse-index when
requested, and do so in t7817.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
The clean_tracked_sparse_directories() method deletes the tracked
directories that go out of scope when the sparse-checkout cone changes,
at least in cone mode. This is new behavior, but is recommended based on
our understanding of how users are interacting with the feature in most
cases.
It is possible that some users will object to the new behavior, so
create a new configuration option 'index.deleteSparseDirectories' that
can be set to 'false' to make clean_tracked_sparse_directories() do
nothing. This will keep all untracked files in the working tree and
cause performance problems with the sparse index, but those trade-offs
are for the user to decide.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
This merges the upstreamable part of the Scalar patches.
Minor merge conflicts (caused by the gvfs-helper) were resolved
trivially.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The 'git maintenance unregister' subcommand has a step that removes the
current repository from the multi-valued maitenance.repo config key.
This fails if the repository is not listed in that key. This makes
running 'git maintenance unregister' twice result in a failure in the
second instance.
Make this task idempotent, since the end result is the same in both
cases: maintenance will no longer run on this repository.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
Teach gvfs-helper to better support the concurrent fetching of the
same packfile by multiple instances.
If 2 instances of gvfs-helper did a POST and requested the same set of
OIDs, they might receive the exact same packfile (same checksum SHA).
Both processes would then race to install their copy of the .pack and
.idx files into the ODB/pack directory.
This is not a problem on Unix (because of filesystem semantics).
On Windows, this can cause an EBUSY/EPERM problem for the loser while
the winner is holding a handle to the target files. (The existing
packfile code already handled simple the existence and/or replacement
case.)
The solution presented here is to silently let the loser claim
victory IIF the .pack and .idx are already present in the ODB.
(We can't check this in advance because we don't know the packfile
SHA checksum until after we receive it and run index-pack.)
We avoid using a per-packfile lockfile (or a single lockfile for
the `vfs-` prefix) to avoid the usual issues with stale lockfiles.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach gvfs-helper to support "/gvfs/prefetch" REST API.
This includes a new `gvfs-helper prefetch --since=<t>` command line option.
And a new `objects.prefetch` verb in `gvfs-helper server` mode.
If `since` argument is omitted, `gvfs-helper` will search the local
shared-cache for the most recent prefetch packfile and start from
there.
The <t> is usually a seconds-since-epoch, but may also be a "friendly"
date -- such as "midnight", "yesterday" and etc. using the existing
date selection mechanism.
Add `gh_client__prefetch()` API to allow `git.exe` to easily call
prefetch (and using the same long-running process as immediate and
queued object fetches).
Expanded t5799 unit tests to include prefetch tests. Test setup now
also builds some commits-and-trees packfiles for testing purposes with
well-known timestamps.
Expanded t/helper/test-gvfs-protocol.exe to support "/gvfs/prefetch"
REST API.
Massive refactor of existing packfile handling in gvfs-helper.c to
reuse more code between "/gvfs/objects POST" and "/gvfs/prefetch".
With this we now properly name packfiles with the checksum SHA1
rather than a date string.
Refactor also addresses some of the confusing tempfile setup and
install_<result> code processing (introduced to handle the ambiguity
of how POST works with commit objects).
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach helper/test-gvfs-protocol to be able to send corrupted
loose blobs.
Add unit test for gvfs-helper to detect receipt of a corrupted loose blob.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Earlier versions of the test always returned a packfile in response to a POST.
Now we look at the number of objects in the POST request.
If > 1, always send a packfile.
If = 1 and it is a commit, send a packfile.
Otherwise, send a loose object.
This is to better model the behavior of the GVFS server/protocol which
treats commits differently.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Create t/helper/test-gvfs-protocol.c and t/t5799-gvfs-helper.sh
to test gvfs-helper.
Create t/helper/test-gvfs-protocol.c as a stand-alone web server that
speaks the GVFS Protocol [1] and serves loose objects and packfiles
to clients. It is borrows heavily from the code in daemon.c.
It includes a "mayhem" mode to cause various network and HTTP errors
to test the retry/recovery ability of gvfs-helper.
Create t/t5799-gvfs-helper.sh to test gvfs-helper.
[1] https://github.com/microsoft/VFSForGit/blob/master/Protocol.md
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Create gvfs-helper. This is a helper tool to use the GVFS Protocol
REST API to fetch objects and configuration data from a GVFS cache-server
or Git server. This tool uses libcurl to send object requests to either
server. This tool creates loose objects and/or packfiles.
Create gvfs-helper-client. This code resides within git proper and
uses the sub-process API to manage gvfs-helper as a long-running background
process.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Teach Git to not throw a fatal error when an explicitly-specified
status-cache file (`git status --deserialize=<foo>`) could not be
found or opened for reading and silently fallback to a traditional
scan.
This matches the behavior when the status-cache file is implicitly
given via a config setting.
Note: the current version causes a test to start failing. Mark this as
an expected result for now.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Verify that `git status --deserialize=x -v` does not crash and
generates the same output as a normal (scanning) status command.
These issues are described in the previous 2 commits.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The fsmonitor script that can be used for running all the git tests
using watchman was causing some of the tests to fail because it wrote
to stderr and created some files for debugging purposes.
Add a new debug script to use with debugging and modify the other script
to remove the code that would cause tests to fail.
Signed-off-by: Kevin Willford <Kevin.Willford@microsoft.com>
With the "--untracked-files=complete" option status computes a
superset of the untracked files. We use this when writing the
status cache. If subsequent deserialize commands ask for either
the complete set or one of the "no", "normal", or "all" subsets,
it can still use the cache file because of filtering in the
deserialize parser.
When running status with the "-uno" option, the long format
status would print a "(use -u to show untracked files)" hint.
When deserializing with the "-uno" option and using a cache computed
with "-ucomplete", the "nothing to commit, working tree clean" message
would be printed instead of the hint.
It was easy to miss because the correct hint message was printed
if the cache was rejected for any reason (and status did the full
fallback).
The "struct wt_status des" structure was initialized with the
content of the status cache (and thus defaulted to "complete").
This change sets "des.show_untracked_files" to the requested
subset from the command-line or config. This allows the long
format to print the hint.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach `git status --deserialize` to either wait indefintely
or immediately fail if the status serialization cache file
is stale.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Fix "git status --deserialize" to correctly report both pathnames
for renames. Add a test case to verify.
A change was made upstream that added an additional "rename_status"
field to the "struct wt_status_change_data" structure. It is used
during the various print routines to decide if 2 pathnames need to
be printed.
5134ccde64
wt-status.c: rename rename-related fields in wt_status_change_data
The fix here is to add that field to the status cache data.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach status deserialize code to reject status cache
when printing in porcelain V2 and there are unresolved
conflicts in the cache file. A follow-on task might
extend the cache format to include this additiona data.
See code for longer explanation.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach status serialization to take an optional pathname on
the command line to direct that cache data be written there
rather than to stdout. When used this way, normal status
results will still be written to stdout.
When no path is given, only binary serialization data is
written to stdout.
Usage:
git status --serialize[=<path>]
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach STATUS to optionally serialize the results of a
status computation to a file.
Teach STATUS to optionally read an existing serialization
file and simply print the results, rather than actually
scanning.
This is intended for immediate status results on extremely
large repos and assumes the use of a service/daemon to
maintain a fresh current status snapshot.
2021-10-30: packet_read() changed its prototype in ec9a37d (pkt-line.[ch]:
remove unused packet_read_line_buf(), 2021-10-14).
2021-10-30: sscanf() now does an extra check that "%d" goes into an "int"
and complains about "uint32_t". Replacing with "%u" fixes the compile-time
error.
2021-10-30: string_list_init() was removed by abf897b (string-list.[ch]:
remove string_list_init() compatibility function, 2021-09-28), so we need to
initialize manually.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
When our patches to support that hook were upstreamed, the hook's name
was eliciting some reviewer suggestions, and it was renamed to
`post-index-change`. These patches (with the new name) made it into
v2.22.0.
However, VFSforGit users may very well have checkouts with that hook
installed under the original name.
To support this, let's just introduce a hack where we look a bit more
closely when we just failed to find the `post-index-change` hook, and
allow any `post-indexchanged` hook to run instead (if it exists).
The vfs does not correctly handle the case when there is a file
that begins with the same prefix as a directory. For example, the
following setup would encounter this issue:
A directory contains a file named `dir1.sln` and a directory
named `dir1/`.
The directory `dir1` contains other files.
The directory `dir1` is in the virtual file system list
The contents of `dir1` should be in the virtual file system, but
it is not. The contents of this directory do not have the skip
worktree bit cleared as expected. The problem is in the
`apply_virtualfilesystem(...)` function where it does not include
the trailing slash of the directory name when looking up the
position in the index to start clearing the skip worktree bit.
This fix is it include the trailing slash when finding the first
index entry from `index_name_pos(...)`.
Add check to see if a directory is included in the virtualfilesystem
before checking the directory hashmap. This allows a directory entry
like foo/ to find all untracked files in subdirectories.
On index load, clear/set the skip worktree bits based on the virtual
file system data. Use virtual file system data to update skip-worktree
bit in unpack-trees. Use virtual file system data to exclude files and
folders not explicitly requested.
Update 2022-04-05: disable the "present-despite-SKIP_WORKTREE" file removal
behavior when 'core.virtualfilesystem' is enabled.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
We found a user who had set "core.gvfs = false" in their global
config. This should not have been necessary, but it also should not
have caused a problem. However, it did.
The reason is that gvfs_load_config_value() is called from config.c
when reading config key/value pairs from all the config files. The
local config should override the global config, and this is done by
config.c reading the global config first then reading the local
config. However, our logic only allowed writing the core_gvfs
variable once.
Put the guards against multiple assignments of core_gvfs into
gvfs_config_is_set() instead, because that will fix the problem
_and_ keep multiple calls to gvfs_config_is_set() from slowing down.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
The following commands and options are not currently supported when working
in a GVFS repo. Add code to detect and block these commands from executing.
1) fsck
2) gc
4) prune
5) repack
6) submodule
8) update-index --split-index
9) update-index --index-version (other than 4)
10) update-index --[no-]skip-worktree
11) worktree
Signed-off-by: Ben Peart <benpeart@microsoft.com>
When using the sparse-checkout feature, the file might not be on disk
because the skip-worktree bit is on.
Signed-off-by: Kevin Willford <kewillf@microsoft.com>
We need to respect that config setting even if we already know that we
have a repository, but have not yet read the config.
The regression test was written by Alejandro Pauly.
2021-10-30: Recent movement of find_hook() into hook.c required moving this
change from run-command.c.
Signed-off-by: Johannes Schindelin <johasc@microsoft.com>
This adds hard-coded call to GVFS.hooks.exe before and after each Git
command runs.
To make sure that this is only called on repositories cloned with GVFS, we
test for the tell-tale .gvfs.
2021-10-30: Recent movement of find_hook() to hook.c required moving these
changes out of run-command.c to hook.c.
Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
If we are going to write an object there is no use in calling
the read object hook to get an object from a potentially remote
source. We would rather just write out the object and avoid the
potential round trip for an object that doesn't exist.
This change adds a flag to the check_and_freshen() and
freshen_loose_object() functions' signatures so that the hook
is bypassed when the functions are called before writing loose
objects. The check for a local object is still performed so we
don't overwrite something that has already been written to one
of the objects directories.
Based on a patch by Kevin Willford.
Signed-off-by: Johannes Schindelin <johasc@microsoft.com>
Hydrate missing loose objects in check_and_freshen() when running
virtualized. Add test cases to verify read-object hook works when
running virtualized.
This hook is called in check_and_freshen() rather than
check_and_freshen_local() to make the hook work also with alternates.
Helped-by: Kevin Willford <kewillf@microsoft.com>
Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
Ensure all filters and EOL conversions are blocked when running under
GVFS so that our projected file sizes will match the actual file size
when it is hydrated on the local machine.
Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>
While performing a fetch with a virtual file system we know that there
will be missing objects and we don't want to download them just because
of the reachability of the commits. We also don't want to download a
pack file with commits, trees, and blobs since these will be downloaded
on demand.
This flag will skip the first connectivity check and by returning zero
will skip the upload pack. It will also skip the second connectivity
check but continue to update the branches to the latest commit ids.
Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Prevent the sparse checkout to delete files that were marked with
skip-worktree bit and are not in the sparse-checkout file.
This is because everything with the skip-worktree bit turned on is being
virtualized and will be removed with the change of HEAD.
There was only one failing test when running with these changes that was
checking to make sure the worktree narrows on checkout which was
expected since we would no longer be narrowing the worktree.
Update 2022-04-05: temporarily set 'sparse.expectfilesoutsideofpatterns' in
test (until we start disabling the "remove present-despite-SKIP_WORKTREE"
behavior with 'core.virtualfilesystem' in a later commit).
Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Derrick Stolee
Johannes Schindelin
Derrick Stolee
Victoria Dye
Lessley Dennington
Jeff Hostetler
Kevin Willford
Jameson Miller
Johannes Schindelin
Kevin Willford
Ben Peart
Alejandro Pauly
Ben Peart