If multiple threads access a directory that is not yet in the cache, the
directory will be loaded by each thread. Only one of the results is added
to the cache, all others are leaked. This wastes performance and memory.
On cache miss, add a future object to the cache to indicate that the
directory is currently being loaded. Subsequent threads register themselves
with the future object and wait. When the first thread has loaded the
directory, it replaces the future object with the result and notifies
waiting threads.
Signed-off-by: Karsten Blees <blees@dcon.de>
Make fscache_enabled() function public rather than static.
Remove unneeded fscache_is_enabled() function.
Change is_fscache_enabled() macro to call fscache_enabled().
is_fscache_enabled() now takes a pathname so that the answer
is more precise and mean "is fscache enabled for this pathname",
since fscache only stores repo-relative paths and not absolute
paths, we can avoid attempting lookups for absolute paths.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach FSCACHE to remember "not found" directories.
This is a performance optimization.
FSCACHE is a performance optimization available for Windows. It
intercepts Posix-style lstat() calls into an in-memory directory
using FindFirst/FindNext. It improves performance on Windows by
catching the first lstat() call in a directory, using FindFirst/
FindNext to read the list of files (and attribute data) for the
entire directory into the cache, and short-cut subsequent lstat()
calls in the same directory. This gives a major performance
boost on Windows.
However, it does not remember "not found" directories. When STATUS
runs and there are missing directories, the lstat() interception
fails to find the parent directory and simply return ENOENT for the
file -- it does not remember that the FindFirst on the directory
failed. Thus subsequent lstat() calls in the same directory, each
re-attempt the FindFirst. This completely defeats any performance
gains.
This can be seen by doing a sparse-checkout on a large repo and
then doing a read-tree to reset the skip-worktree bits and then
running status.
This change reduced status times for my very large repo by 60%.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Checking the work tree status is quite slow on Windows, due to slow lstat
emulation (git calls lstat once for each file in the index). Windows
operating system APIs seem to be much better at scanning the status
of entire directories than checking single files.
Add an lstat implementation that uses a cache for lstat data. Cache misses
read the entire parent directory and add it to the cache. Subsequent lstat
calls for the same directory are served directly from the cache.
Also implement opendir / readdir / closedir so that they create and use
directory listings in the cache.
The cache doesn't track file system changes and doesn't plug into any
modifying file APIs, so it has to be explicitly enabled for git functions
that don't modify the working copy.
Note: in an earlier version of this patch, the cache was always active and
tracked file system changes via ReadDirectoryChangesW. However, this was
much more complex and had negative impact on the performance of modifying
git commands such as 'git checkout'.
Signed-off-by: Karsten Blees <blees@dcon.de>
Teach read_directory_recursive() and add_excludes() to
be aware of optional fscache and avoid trying to open()
and fstat() non-existant ".gitignore" files in every
directory in the worktree.
The current code in add_excludes() calls open() and then
fstat() for a ".gitignore" file in each directory present
in the worktree. Change that when fscache is enabled to
call lstat() first and if present, call open().
This seems backwards because both lstat needs to do more
work than fstat. But when fscache is enabled, fscache will
already know if the .gitignore file exists and can completely
avoid the IO calls. This works because of the lstat diversion
to mingw_lstat when fscache is enabled.
This reduced status times on a 350K file enlistment of the
Windows repo on a NVMe SSD by 0.25 seconds.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Add a macro to mark code sections that only read from the file system,
along with a config option and documentation.
This facilitates implementation of relatively simple file system level
caches without the need to synchronize with the file system.
Enable read-only sections for 'git status' and preload_index.
Signed-off-by: Karsten Blees <blees@dcon.de>
Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite
slow. Windows operating system APIs seem to be much better at scanning the
status of entire directories than checking single files. A caching
implementation may improve performance by bulk-reading entire directories
or reusing data obtained via opendir / readdir.
Make the lstat implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Emulating the POSIX dirent API on Windows via FindFirstFile/FindNextFile is
pretty staightforward, however, most of the information provided in the
WIN32_FIND_DATA structure is thrown away in the process. A more
sophisticated implementation may cache this data, e.g. for later reuse in
calls to lstat.
Make the dirent implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Define a base DIR structure with pointers to readdir/closedir that match
the opendir implementation (i.e. similar to vtable pointers in OOP).
Define readdir/closedir so that they call the function pointers in the DIR
structure. This allows to choose the opendir implementation on a
call-by-call basis.
Move the fixed sized dirent.d_name buffer to the dirent-specific DIR
structure, as d_name may be implementation specific (e.g. a caching
implementation may just set d_name to point into the cache instead of
copying the entire file name string).
Signed-off-by: Karsten Blees <blees@dcon.de>
This fixes https://github.com/msysgit/git/issues/182.
Inspired by Pull Request 218 using code from @PhilipDavis.
[jes: simplified code quite a bit]
Signed-off-by: Kelly Heller <kkheller@cedrus.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We will use them in the upcoming "FSCache" patches (to accelerate
sequential lstat() calls).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Turns out that it don't work so well on Vista, see
https://github.com/git-for-windows/git/issues/1742 for details.
According to https://devblogs.microsoft.com/oldnewthing/?p=8873, it
*should* work on Windows Vista and later.
But apparently there are issues on Windows Vista when pipes are
involved. Given that Windows Vista is past its end of life (official
support ended on April 11th, 2017), let's not spend *too* much time on
this issue and just disable the file handle inheritance restriction on
any Windows version earlier than Windows 7.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
By default, CreateProcess() does not inherit any open file handles,
unless the bInheritHandles parameter is set to TRUE. Which we do need to
set because we need to pass in stdin/stdout/stderr to talk to the child
processes. Sadly, this means that all file handles (unless marked via
O_NOINHERIT) are inherited.
This lead to problems in GVFS Git, where a long-running read-object hook
is used to hydrate missing objects, and depending on the circumstances,
might only be called *after* Git opened a file handle.
Ideally, we would not open files without O_NOINHERIT unless *really*
necessary (i.e. when we want to pass the opened file handle as standard
handle into a child process), but apparently it is all-too-easy to
introduce incorrect open() calls: this happened, and prevented updating
a file after the read-object hook was started because the hook still
held a handle on said file.
Happily, there is a solution: as described in the "Old New Thing"
https://blogs.msdn.microsoft.com/oldnewthing/20111216-00/?p=8873 there
is a way, starting with Windows Vista, that lets us define precisely
which handles should be inherited by the child process.
And since we bumped the minimum Windows version for use with Git for
Windows to Vista with v2.10.1 (i.e. a *long* time ago), we can use this
method. So let's do exactly that.
We need to make sure that the list of handles to inherit does not
contain duplicates; Otherwise CreateProcessW() would fail with
ERROR_INVALID_ARGUMENT.
While at it, stop setting errno to ENOENT unless it really is the
correct value.
Also, fall back to not limiting handle inheritance under certain error
conditions (e.g. on Windows 7, which is a lot stricter in what handles
you can specify to limit to).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
For some reason, when being called via TortoiseGit the standard handles,
or at least what is returned by _get_osfhandle(0) for standard input,
can take on the value (HANDLE)-2 (which is not a legal value, according
to the documentation).
Even if this value is not documented anywhere, CreateProcess() seems to
work fine without complaints if hStdInput set to this value.
In contrast, the upcoming code to restrict which file handles get
inherited by spawned processes would result in `ERROR_INVALID_PARAMETER`
when including such handle values in the list.
To help this, special-case the value (HANDLE)-2 returned by
_get_osfhandle() and replace it with INVALID_HANDLE_VALUE, which will
hopefully let the handle inheritance restriction work even when called
from TortoiseGit.
This fixes https://github.com/git-for-windows/git/issues/1481
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When spawning child processes, we really should be careful which file
handles we let them inherit.
This is doubly important on Windows, where we cannot rename, delete, or
modify files if there is still a file handle open.
Sadly, we have to guard this test inside #ifdef WIN32: we need to use
the value of the HANDLE directly, and that concept does not exist on
Linux/Unix.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These changes are necessary to support better Git for Windows' new
auto-update feature.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch addresses the bug where Git for Windows 2.x' Git GUI
failed to generate a working shortcut via Repository>Create Desktop
Shortcut.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When using remotes (with git-flow especially), the remote reference names
are almost always wordwrapped in the "list references" window because it's
somewhat narrow by default. It's possible to resize it with a mouse,
but it's annoying to have to do this every time, especially on Windows 10,
where the window border seems to be only one (1) pixel wide, thus making
the grabbing of the window border tricky.
Signed-off-by: James J. Raden <james.raden@gmail.com>
git-gui tries to temporary set GIT_DIR for starting gitk and restore
it back after they are started. But in case of GIT_DIR which was not set
prior to invocation it is not unset after it. This affects commands
which can be later started from that git gui, for example "Git Bash".
Fix it.
Signed-off-by: Max Kirillov <max@max630.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Since v2.9.0, Git knows about the config variable core.hookspath
that allows overriding the path to the directory containing the
Git hooks.
Since v2.10.0, the `--git-path` option respects that config
variable, too, so we may just as well use that command.
For Git versions older than v2.5.0 (which was the first version to
support the `--git-path` option for the `rev-parse` command), we
simply fall back to the previous code.
This fixes https://github.com/git-for-windows/git/issues/1755
Initial-patch-by: Philipp Gortan <philipp@gortan.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git for Windows 2.x ships with an executable that starts the Git Bash
with all the environment variables and what not properly set up. It is
also adjusted according to the Terminal emulator option chosen when
installing Git for Windows (while `bash.exe --login -i` would always
launch with Windows' default console).
So let's use that executable (usually C:\Program Files\Git\git-bash.exe)
instead of `bash.exe --login -i` if its presence was detected.
This fixes https://github.com/git-for-windows/git/issues/490
Signed-off-by: Thomas Kläger <thomas.klaeger@10a.ch>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Tcl/Tk 8.6 introduced new events for the cursor left/right keys and
apparently changed the behavior of the previous event.
Let's work around that by using the new events when we are running with
Tcl/Tk 8.6 or later.
This fixes https://github.com/git-for-windows/git/issues/495
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
"Question?" is maybe not the most informative thing to ask. In the
absence of better information, it is the best we can do, of course.
However, Git for Windows' auto updater just learned the trick to use
git-gui--askyesno to ask the user whether to update now or not. And in
this scripted scenario, we can easily pass a command-line option to
change the window title.
So let's support that with the new `--title <title>` option.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git for Windows now ships with the new Git icon from git-scm.com. Use that
icon file if it exists instead of the old procedurally drawn one.
This patch was sent upstream but so far no decision on its inclusion was
made, so commit it to our fork.
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
The text wrapping seems to be aligned to the right side of the Yes
button, leaving an awful lot of empty space.
Let's try to counter this by using pixel units.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Assumes file names in git tree objects are UTF-8 encoded.
On most unix systems, the system encoding (and thus the TCL system
encoding) will be UTF-8, so file names will be displayed correctly.
On Windows, it is impossible to set the system encoding to UTF-8.
Changing the TCL system encoding (via 'encoding system ...', e.g. in the
startup code) is explicitly discouraged by the TCL docs.
Change gitk functions dealing with file names to always convert
from and to UTF-8.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In MSYS2, we have two Python interpreters at our disposal, so we can
include the Python stuff in the build.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Make use of the new environment variable GIT_ASK_YESNO to support the
recently implemented fallback in case unlink, rename or rmdir fail for
files in use on Windows. The added dialog will present a yes/no question
to the the user which will currently be used by the windows compat layer
to let the user retry a failed file operation.
Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
Let's use the convenient lazy loading functions in winansi to
dynamically load a function that older Windows versions might not have.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
During a `rebase -r`, when a `merge -c` is executed that can
fast-forward, we *still* want to edit the commit message. Let's force
this *not* to fast-forward instead.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Fix racy fsmonitor
The `t7519-status-fsmonitor.sh` tests became a *lot* more flaky with the
recent fsmonitor fix (`js/fsmonitor-refresh-after-discarding-index`).
That fix, however, did not introduce the flakiness, but it just made it
much more likely to be hit. And it seemed to be hit *only* on Windows.
The reason, though, is that the fsmonitor feature failed to mark the
in-memory index as changed, i.e. in need of writing, and it was the
`has_racy_timestamp()` test that hid this bug in most cases (although a
lot less on Windows, where the files' mtimes are actually a lot more
accurate than on Linux).
This fixes https://github.com/gitgitgadget/git/issues/197
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch cleans up some left-overs that were forgotten when
removing the scripted `git rebase`.
As these patches are based on top of v2.22.0-rc1 (which *did* drop the
scripted `git-rebase.sh`), instead of v2.21.0 (on which the current `master`
of Git for Windows is based, and which did *not* yet drop the scripted
`git rebase`) it does not make sense to try to backport them to
`master`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch is a backport of
https://github.com/gitgitgadget/git/pull/208, which avoids a lock
contention in `git gc --auto` where the `git gc` process holds a read
lock to the `commit-graph` file (if `core.commitgraph=true`) and the
spawned `git commit-graph write` (if `gc.writecommitgraph=true`) tries
to overwrite it.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In Git for Windows, there is an option to mark the .git directory as
hidden. Our test cases verify this by using the system utility
`attrib.exe`.
This file name is unfortunately quite generic, and overlaps with a
Unix-y utility that might be hiding the system one in the `PATH`.
Let's specify explicitly which `attrib` to use.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
After writing to `stdout` and before reading from `stdin`, it is a good
idea to flush the former.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It is a good idea in general to avoid pipes in test cases, as it makes
things more debuggable to have files to inspect (instead of ephemereal
piped data that is long gone).
This also seemed to work around a problem where MSYS2' Perl would
segfault which may, or may not, still be present today.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Johannes Schindelin
Karsten Blees
Jeff Hostetler
Kelly Heller
James J. Raden
Max Kirillov
Thomas Kläger
Sebastian Schuberth
Heiko Voigt
Junio C Hamano
Johannes Schindelin