With Windows 10 Build 14972 in Developer Mode, a new flag is supported
by CreateSymbolicLink() to create symbolic links even when running
outside of an elevated session (which was previously required).
This new flag is called SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE and
has the numeric value 0x02.
Previous Windows 10 versions will not understand that flag and return an
ERROR_INVALID_PARAMETER, therefore we have to be careful to try passing
that flag only when the build number indicates that it is supported.
For more information about the new flag, see this blog post:
https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/
This patch is loosely based on the patch submitted by Samuel D. Leslie
as https://github.com/git-for-windows/git/pull/1184.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Symlinks on Windows have a flag that indicates whether the target is a file
or a directory. Symlinks of wrong type simply don't work. This even affects
core Win32 APIs (e.g. DeleteFile() refuses to delete directory symlinks).
However, CreateFile() with FILE_FLAG_BACKUP_SEMANTICS doesn't seem to care.
Check the target type by first creating a tentative file symlink, opening
it, and checking the type of the resulting handle. If it is a directory,
recreate the symlink with the directory flag set.
It is possible to create symlinks before the target exists (or in case of
symlinks to symlinks: before the target type is known). If this happens,
create a tentative file symlink and postpone the directory decision: keep
a list of phantom symlinks to be processed whenever a new directory is
created in mingw_mkdir().
Limitations: This algorithm may fail if a link target changes from file to
directory or vice versa, or if the target directory is created in another
process.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Implement symlink() that always creates file symlinks. Fails with ENOSYS
if symlinks are disabled or unsupported.
Note: CreateSymbolicLinkW() was introduced with symlink support in Windows
Vista. For compatibility with Windows XP, we need to load it dynamically
and fail gracefully if it isnt's available.
Signed-off-by: Karsten Blees <blees@dcon.de>
Implement readlink() by reading NTFS reparse points. Works for symlinks
and directory junctions. If symlinks are disabled, fail with ENOSYS.
Signed-off-by: Karsten Blees <blees@dcon.de>
If symlinks are enabled, resolve all symlinks when changing directories,
as required by POSIX.
Note: Git's real_path() function bases its link resolution algorithm on
this property of chdir(). Unfortunately, the current directory on Windows
is limited to only MAX_PATH (260) characters. Therefore using symlinks and
long paths in combination may be problematic.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
MSVCRT's _wrename() cannot rename symlinks over existing files: it returns
success without doing anything. Newer MSVCR*.dll versions probably do not
have this problem: according to CRT sources, they just call MoveFileEx()
with the MOVEFILE_COPY_ALLOWED flag.
Get rid of _wrename() and call MoveFileEx() with proper error handling.
Signed-off-by: Karsten Blees <blees@dcon.de>
_wunlink() / DeleteFileW() refuses to delete symlinks to directories. If
_wunlink() fails with ERROR_ACCESS_DENIED, try _wrmdir() as well.
Signed-off-by: Karsten Blees <blees@dcon.de>
Symlinks on Windows don't work the same way as on Unix systems. E.g. there
are different types of symlinks for directories and files, creating
symlinks requires administrative privileges etc.
By default, disable symlink support on Windows. I.e. users explicitly have
to enable it with 'git config [--system|--global] core.symlinks true'.
The test suite ignores system / global config files. Allow testing *with*
symlink support by checking if native symlinks are enabled in MSys2 (via
'MSYS=winsymlinks:nativestrict').
Reminder: This would need to be changed if / when we find a way to run the
test suite in a non-MSys-based shell (e.g. dash).
Signed-off-by: Karsten Blees <blees@dcon.de>
The retry pattern is duplicated in three places. It also seems to be too
hard to use: mingw_unlink() and mingw_rmdir() duplicate the code to retry,
and both of them do so incompletely. They also do not restore errno if the
user answers 'no'.
Introduce a retry_ask_yes_no() helper function that handles retry with
small delay, asking the user, and restoring errno.
mingw_unlink: include _wchmod in the retry loop (which may fail if the
file is locked exclusively).
mingw_rmdir: include special error handling in the retry loop.
Signed-off-by: Karsten Blees <blees@dcon.de>
Git typically doesn't trust the stat.st_size member of symlinks (e.g. see
strbuf_readlink()). However, some functions take shortcuts if st_size is 0
(e.g. diff_populate_filespec()).
In mingw_lstat() and fscache_lstat(), make sure to return an adequate size.
The extra overhead of opening and reading the reparse point to calculate
the exact size is not necessary, as git doesn't rely on the value anyway.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Move S_IFLNK detection to file_attr_to_st_mode() and reuse it in fscache.
Implement DT_LNK detection in dirent.c and the fscache readdir version.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When obtaining lstat information for reparse points, we need to call
FindFirstFile() in addition to GetFileInformationEx() to obtain the type
of the reparse point (symlink, mount point etc.). However, currently there
is no error handling whatsoever if FindFirstFile() fails.
Call FindFirstFile() before modifying the stat *buf output parameter and
error out if the call fails.
Note: The FindFirstFile() return value includes all the data that we get
from GetFileAttributesEx(), so we could replace GetFileAttributesEx() with
FindFirstFile(). We don't do that because GetFileAttributesEx() is about
twice as fast for single files. I.e. we only pay the extra cost of calling
FindFirstFile() in the rare case that we encounter a reparse point.
Note: The indentation of the remaining reparse point code will be fixed in
the next patch.
Signed-off-by: Karsten Blees <blees@dcon.de>
With the new mingw_stat() implementation, do_lstat() is only called from
mingw_lstat() (with follow == 0). Remove the extra function and the old
mingw_stat()-specific (follow == 1) logic.
Signed-off-by: Karsten Blees <blees@dcon.de>
With respect to symlinks, the current stat() implementation is almost the
same as lstat(): except for the file type (st_mode & S_IFMT), it returns
information about the link rather than the target.
Implement stat by opening the file with as little permissions as possible
and calling GetFileInformationByHandle on it. This way, all link resoltion
is handled by the Windows file system layer.
If symlinks are disabled, use lstat() as before, but fail with ELOOP if a
symlink would have to be resolved.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
GetFileAttributes cannot handle paths with trailing dir separator. The
current [l]stat implementation calls GetFileAttributes twice if the path
has trailing slashes (first with the original path passed to [l]stat, and
and a second time with a path copy with trailing '/' removed).
With Unicode conversion, we get the length of the path for free and also
have a (wide char) buffer that can be modified.
Remove trailing directory separators before calling the Win32 API.
Signed-off-by: Karsten Blees <blees@dcon.de>
strbuf_readlink() refuses to read link targets that exceed PATH_MAX (even
if a sufficient size was specified by the caller).
As some platforms support longer paths, remove this restriction (similar
to strbuf_getcwd()).
Signed-off-by: Karsten Blees <blees@dcon.de>
strbuf_readlink() calls readlink() twice if the hint argument specifies the
exact size of the link target (e.g. by passing stat.st_size as returned by
lstat()). This is necessary because 'readlink(..., hint) == hint' could
mean that the buffer was too small.
Use hint + 1 as buffer size to prevent this.
Signed-off-by: Karsten Blees <blees@dcon.de>
These patches are probably no longer necessary. Make sure before
dropping them, though.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
A change between versions 2.4.1 and 2.6.0 of the MSYS2 runtime modified
how Cygwin's runtime (and hence Git for Windows' MSYS2 runtime
derivative) handles locales: d16a56306d (Consolidate wctomb/mbtowc calls
for POSIX-1.2008, 2016-07-20).
An unintended side-effect is that "cold-calling" into the POSIX
emulation will start with a locale based on the current code page,
something that Git for Windows is very ill-prepared for, as it expects
to be able to pass a command-line containing non-ASCII characters to the
shell without having those characters munged.
One symptom of this behavior: when `git clone` or `git fetch` shell out
to call `git-upload-pack` with a path that contains non-ASCII
characters, the shell tried to interpret the entire command-line
(including command-line parameters) as executable path, which obviously
must fail.
This fixes https://github.com/git-for-windows/git/issues/1036
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
POSIX-to-Windows path mangling would make it fail. Symptoms:
++ init_git
++ rm -fr .git
++ git init
Initialized empty Git repository in [...]
++ git remote add svnsim testsvn::sim:///usr/src/git/wip5/t/t9154/svn.dump
++ git remote add svnfile testsvn::file:///usr/src/git/wip5/t/t9154/svn.dump
++ git fetch svnsim
progress Imported commit 1.
fatal: Write to frontend failed: Bad file descriptor
fast-import: dumping crash report to .git/fast_import_crash_23356
fatal: error while running fast-import
fatal: unexpected end of fast-import feedback
error: last command exited with $?=128
not ok 1 - simple fetch
Since the remote-svn project seems to be dormant at the moment (and not
complete enough to be used, which is a pity), let's just skip this test
on Windows.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
With the recent update in efee955 (gpg-interface: check gpg signature
creation status, 2016-06-17), we ask GPG to send all status updates to
stderr, and then catch the stderr in an strbuf.
But GPG might fail, and send error messages to stderr. And we simply
do not show them to the user.
Even worse: this swallows any interactive prompt for a passphrase. And
detaches stderr from the tty so that the passphrase cannot be read.
So while the first problem could be fixed (by printing the captured
stderr upon error), the second problem cannot be easily fixed, and
presents a major regression.
So let's just revert commit efee9553a4.
This fixes https://github.com/git-for-windows/git/issues/871
Cc: Michael J Gruber <git@drmicha.warpmail.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
To support Git Bash running in a MinTTY, we use a dirty trick to access
the MSYS2 pseudo terminal: we execute a Bash snippet that accesses
/dev/tty.
The idea was to fall back to writing to/reading from CONOUT$/CONIN$ if
that Bash call failed because Bash was not found.
However, we should fall back even in other error conditions, because we
have not successfully read the user input. Let's make it so.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Accessing the Windows console through the special CONIN$ / CONOUT$ devices
doesn't work properly for non-ASCII usernames an passwords.
It also doesn't work for terminal emulators that hide the native console
window (such as mintty), and 'TERM=xterm*' is not necessarily a reliable
indicator for such terminals.
The new shell_prompt() function, on the other hand, works fine for both
MSys1 and MSys2, in native console windows as well as mintty, and properly
supports Unicode. It just needs bash on the path (for 'read -s', which is
bash-specific).
On Windows, try to use the shell to read from the terminal. If that fails
with ENOENT (i.e. bash was not found), use CONIN/OUT as fallback.
Note: To test this, create a UTF-8 credential file with non-ASCII chars,
e.g. in git-bash: 'echo url=http://täst.com > cred.txt'. Then in git-cmd,
'git credential fill <cred.txt' works (shell version), while calling git
without the git-wrapper (i.e. 'mingw64\bin\git credential fill <cred.txt')
mangles non-ASCII chars in both console output and input.
Signed-off-by: Karsten Blees <blees@dcon.de>
The `git_terminal_prompt()` function expects the terminal window to be
attached to a Win32 Console. However, this is not the case with terminal
windows other than `cmd.exe`'s, e.g. with MSys2's own `mintty`.
Non-cmd terminals such as `mintty` still have to have a Win32 Console
to be proper console programs, but have to hide the Win32 Console to
be able to provide more flexibility (such as being resizeable not only
vertically but also horizontally). By writing to that Win32 Console,
`git_terminal_prompt()` manages only to send the prompt to nowhere and
to wait for input from a Console to which the user has no access.
This commit introduces a function specifically to support `mintty` -- or
other terminals that are compatible with MSys2's `/dev/tty` emulation. We
use the `TERM` environment variable as an indicator for that: if the value
starts with "xterm" (such as `mintty`'s "xterm_256color"), we prefer to
let `xterm_prompt()` handle the user interaction.
The most prominent user of `git_terminal_prompt()` is certainly
`git-remote-https.exe`. It is an interesting use case because both
`stdin` and `stdout` are redirected when Git calls said executable, yet
it still wants to access the terminal.
When running inside a `mintty`, the terminal is not accessible to the
`git-remote-https.exe` program, though, because it is a MinGW program
and the `mintty` terminal is not backed by a Win32 console.
To solve that problem, we simply call out to the shell -- which is an
*MSys2* program and can therefore access `/dev/tty`.
Helped-by: nalla <nalla@hamal.uberspace.de>
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
A ton of Git commands simply do not read (or at least parse) the core.*
settings. This is not good, as Git for Windows relies on the
core.longPaths setting to be read quite early on.
So let's just make sure that all commands read the config and give
platform_core_config() a chance.
This patch teaches tons of Git commands to respect the config setting
`core.longPaths = true`, including `pack-refs`, thereby fixing
https://github.com/git-for-windows/git/issues/1218
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).
Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.
Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).
Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).
Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).
Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.
Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.
While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.
Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199
[jes: adjusted test number to avoid conflicts, added support for
chdir(), etc]
Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
There is a problem in the way 9ac3f0e5b3 (pack-objects: fix
performance issues on packing large deltas, 2018-07-22) initializes that
mutex in the `packing_data` struct. The problem manifests in a
segmentation fault on Windows, when a mutex (AKA critical section) is
accessed without being initialized. (With pthreads, you apparently do
not really have to initialize them?)
This was reported in https://github.com/git-for-windows/git/issues/1839.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Using FindFirstFileExW() requires the OS to allocate a 64K buffer for each
directory and then free it when we call FindClose(). Update fscache to call
the underlying kernel API NtQueryDirectoryFile so that we can do the buffer
management ourselves. That allows us to allocate a single buffer for the
lifetime of the cache and reuse it for each directory.
This change improves performance of 'git status' by 18% in a repo with ~200K
files and 30k folders.
Documentation for NtQueryDirectoryFile can be found at:
https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/ntifs/nf-ntifs-ntquerydirectoryfilehttps://docs.microsoft.com/en-us/windows/desktop/FileIO/file-attribute-constantshttps://docs.microsoft.com/en-us/windows/desktop/fileio/reparse-point-tags
To determine if the specified directory is a symbolic link, inspect the
FileAttributes member to see if the FILE_ATTRIBUTE_REPARSE_POINT flag is
set. If so, EaSize will contain the reparse tag (this is a so far
undocumented feature, but confirmed by the NTFS developers). To
determine if the reparse point is a symbolic link (and not some other
form of reparse point), test whether the tag value equals the value
IO_REPARSE_TAG_SYMLINK.
The NtQueryDirectoryFile() call works best (and on Windows 8.1 and
earlier, it works *only*) with buffer sizes up to 64kB. Which is 32k
wide characters, so let's use that as our buffer size.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The recent change to make fscache thread specific relied on fscache_enable()
being called first from the primary thread before being called in parallel
from worker threads. Make that more robust and protect it with a critical
section to avoid any issues.
Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Now that the fscache is single threaded, take advantage of the mem_pool as
the allocator to significantly reduce the cost of allocations and frees.
With the reduced cost of free, in future patches, we can start freeing the
fscache at the end of commands instead of just leaking it.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
The threading model for fscache has been to have a single, global cache.
This puts requirements on it to be thread safe so that callers like
preload-index can call it from multiple threads. This was implemented
with a single mutex and completion events which introduces contention
between the calling threads.
Simplify the threading model by making fscache thread specific. This allows
us to remove the global mutex and synchronization events entirely and instead
associate a fscache with every thread that requests one. This works well with
the current multi-threading which divides the cache entries into blocks with
a separate thread processing each block.
At the end of each worker thread, if there is a fscache on the primary
thread, merge the cached results from the worker into the primary thread
cache. This enables us to reuse the cache later especially when scanning for
untracked files.
In testing, this reduced the time spent in preload_index() by about 25% and
also reduced the CPU utilization significantly. On a repo with ~200K files,
it reduced overall status times by ~12%.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Update enable_fscache() to take an optional initial size parameter which is
used to initialize the hashmap so that it can avoid having to rehash as
additional entries are added.
Add a separate disable_fscache() macro to make the code clearer and easier
to read.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Add tracing around initializing and discarding mempools. In discard report
on the amount of memory unused in the current block to help tune setting
the initial_size.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Track fscache hits and misses for lstat and opendir requests. Reporting of
statistics is done when the cache is disabled for the last time and freed
and is only reported if GIT_TRACE_FSCACHE is set.
Sample output is:
11:33:11.836428 compat/win32/fscache.c:433 fscache: lstat 3775, opendir 263, total requests/misses 4052/269
Signed-off-by: Ben Peart <benpeart@microsoft.com>
At the end of the status command, disable and free the fscache so that we
don't leak the memory and so that we can dump the fscache statistics.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Use FindFirstFileExW with FindExInfoBasic to avoid forcing NTFS to look up
the short name. Also switch to a larger (64K vs 4K) buffer using
FIND_FIRST_EX_LARGE_FETCH to minimize round trips to the kernel.
In a repo with ~200K files, this drops warm cache status times from 3.19
seconds to 2.67 seconds for a 16% savings.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
On file systems that support it, this can dramatically speed up operations
like add, commit, describe, rebase, reset, rm that would otherwise have to
lstat() every file to "re-match" the stat information in the index to that
of the file system.
On a synthetic repo with 1M files, "git reset" dropped from 52.02 seconds to
14.42 seconds for a savings of 72%.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
This is retry of #1419.
I added flush_fscache macro to flush cached stats after disk writing
with tests for regression reported in #1438 and #1442.
git checkout checks each file path in sorted order, so cache flushing does not
make performance worse unless we have large number of modified files in
a directory containing many files.
Using chromium repository, I tested `git checkout .` performance when I
delete 10 files in different directories.
With this patch:
TotalSeconds: 4.307272
TotalSeconds: 4.4863595
TotalSeconds: 4.2975562
Avg: 4.36372923333333
Without this patch:
TotalSeconds: 20.9705431
TotalSeconds: 22.4867685
TotalSeconds: 18.8968292
Avg: 20.7847136
I confirmed this patch passed all tests in t/ with core_fscache=1.
Signed-off-by: Takuto Ikuta <tikuta@chromium.org>
When I do git fetch, git call file stats under .git/objects for each
refs. This takes time when there are many refs.
By enabling fscache, git takes file stats by directory traversing and that
improved the speed of fetch-pack for repository having large number of
refs.
In my windows workstation, this improves the time of `git fetch` for
chromium repository like below. I took stats 3 times.
* With this patch
TotalSeconds: 9.9825165
TotalSeconds: 9.1862075
TotalSeconds: 10.1956256
Avg: 9.78811653333333
* Without this patch
TotalSeconds: 15.8406702
TotalSeconds: 15.6248053
TotalSeconds: 15.2085938
Avg: 15.5580231
Signed-off-by: Takuto Ikuta <tikuta@chromium.org>
Fix regression described in:
https://github.com/git-for-windows/git/issues/1392
which was introduced in:
b2353379bb
Problem Symptoms
================
When the user has a .gitignore file that is a symlink, the fscache
optimization introduced above caused the stat-data from the symlink,
rather that of the target file, to be returned. Later when the ignore
file was read, the buffer length did not match the stat.st_size field
and we called die("cannot use <path> as an exclude file")
Optimization Rationale
======================
The above optimization calls lstat() before open() primarily to ask
fscache if the file exists. It gets the current stat-data as a side
effect essentially for free (since we already have it in memory).
If the file does not exist, it does not need to call open(). And
since very few directories have .gitignore files, we can greatly
reduce time spent in the filesystem.
Discussion of Fix
=================
The above optimization calls lstat() rather than stat() because the
fscache only intercepts lstat() calls. Calls to stat() stay directed
to the mingw_stat() completly bypassing fscache. Furthermore, calls
to mingw_stat() always call {open, fstat, close} so that symlinks are
properly dereferenced, which adds *additional* open/close calls on top
of what the original code in dir.c is doing.
Since the problem only manifests for symlinks, we add code to overwrite
the stat-data when the path is a symlink. This preserves the effect of
the performance gains provided by the fscache in the normal case.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Make fscache_enabled() function public rather than static.
Remove unneeded fscache_is_enabled() function.
Change is_fscache_enabled() macro to call fscache_enabled().
is_fscache_enabled() now takes a pathname so that the answer
is more precise and mean "is fscache enabled for this pathname",
since fscache only stores repo-relative paths and not absolute
paths, we can avoid attempting lookups for absolute paths.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach read_directory_recursive() and add_excludes() to
be aware of optional fscache and avoid trying to open()
and fstat() non-existant ".gitignore" files in every
directory in the worktree.
The current code in add_excludes() calls open() and then
fstat() for a ".gitignore" file in each directory present
in the worktree. Change that when fscache is enabled to
call lstat() first and if present, call open().
This seems backwards because both lstat needs to do more
work than fstat. But when fscache is enabled, fscache will
already know if the .gitignore file exists and can completely
avoid the IO calls. This works because of the lstat diversion
to mingw_lstat when fscache is enabled.
This reduced status times on a 350K file enlistment of the
Windows repo on a NVMe SSD by 0.25 seconds.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach "add" to use preload-index and fscache features
to improve performance on very large repositories.
During an "add", a call is made to run_diff_files()
which calls check_remove() for each index-entry. This
calls lstat(). On Windows, the fscache code intercepts
the lstat() calls and builds a private cache using the
FindFirst/FindNext routines, which are much faster.
Somewhat independent of this, is the preload-index code
which distributes some of the start-up costs across
multiple threads.
We need to keep the call to read_cache() before parsing the
pathspecs (and hence cannot use the pathspecs to limit any preload)
because parse_pathspec() is using the index to determine whether a
pathspec is, in fact, in a submodule. If we would not read the index
first, parse_pathspec() would not error out on a path that is inside
a submodule, and t7400-submodule-basic.sh would fail with
not ok 47 - do not add files from a submodule
We still want the nice preload performance boost, though, so we simply
call read_cache_preload(&pathspecs) after parsing the pathspecs.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Bert Belder
Johannes Schindelin
Karsten Blees
Doug Kelly
Ben Peart
Takuto Ikuta
Jeff Hostetler