When tests are run in parallel and a few tests fail, it does not help
that the output of the terminal is totally confusing, as you rarely know
which test which line came from.
So introduce the option '--tee' which triggers that the output of the
tests will be written to t/test-results/$TEST.out in addition to the
terminal, where $TEST is the basename of the script.
Unfortunately, there seems to be no way to redirect a given file
descriptor to a specified subprocess in POSIX shell, only redirection
to a file is supported via 'exec > $FILE'.
At least with bash, one might think that 'exec >($COMMAND)' would work
as intended, but it does not.
The common way to work around the lack of proper tools support is to
work with named pipes, alas, one of our most beloved platforms does not
really support named pipes. Besides, we would need a pipe for every
script, as the whole point of this patch is to allow parallel execution.
Therefore, we handle the redirection in the following way: when '--tee'
was passed to the test script, the variable GIT_TEST_TEE_STARTED is set
(to avoid triggering that code path again) and the script is started
_again_, in a subshell, redirected to the command "tee".
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This patch makes --valgrind try to override _all_ Git binaries in the
PATH, and it makes it an error to call *.sh and *.perl scripts directly.
While it is not strictly necessary to look through the whole PATH to
find git binaries to override, it is in line with running an expensive
test (which valgrind is) to make extra sure that only binaries are
tested that actually come from the git.git checkout.
In the same spirit, we can test that neither our test suite nor our
scripts try to run the *.sh or *.perl scripts directly.
It's more like a "because we can" than a "this is tightly connected
to valgrind", but in the author's opinion "because we can" is "so we
should" in this case.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This patch adds the ability to use valgrind's memcheck tool to
diagnose memory problems in Git while running the test scripts.
It requires valgrind 3.4.0 or newer.
It works by creating symlinks to a valgrind script, which have the same
name as our Git binaries, and then putting that directory in front of
the test script's PATH as well as set GIT_EXEC_PATH to that directory.
Git scripts are symlinked from that directory directly.
That way, Git binaries called by Git scripts are valgrinded, too.
Valgrind can be used by specifying "GIT_TEST_OPTS=--valgrind" in the
make invocation. Any invocation of git that finds any errors under
valgrind will exit with failure code 126. Any valgrind output will go
to the usual stderr channel for tests (i.e., /dev/null, unless -v has
been specified).
If you need to pass options to valgrind -- you might want to run
another tool than memcheck, for example -- you can set the environment
variable GIT_VALGRIND_OPTIONS.
A few default suppressions are included, since libz seems to trigger
quite a few false positives. We'll assume that libz works and that we
can ignore any errors which are reported there.
Note: it is safe to run the valgrind tests in parallel, as the links in
t/valgrind/bin/ are created using proper locking.
Initial patch and all the hard work by Jeff King.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Brian Gernhardt noticed that t3411 was broken recently on case insensitive
filesystems.
0088496 (test-lib.sh: introduce test_commit() and test_merge() helpers,
2009-01-27) used a tag and a file with the same name, only different in
case, and converted many existing tests that needed only a file (or a
tag).
Some tests may want to refer to a rev or a file, but on a filesystem that
loses cases, referring to either without disambiguation mark ("--") on the
command line now triggers an error (t3411 was the only one such test).
Fix it by using a filename that is different from the tagname each step
creates.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Often we just need to add a commit with a given (short) name, that will
be tagged with the same name. Now, relatively complicated graphs can be
constructed easily and in a clear fashion:
test_commit A &&
test_commit B &&
git checkout A &&
test_commit C &&
test_merge D B
will construct this graph:
A - B
\ \
C - D
For simplicity, files whose name is the lower case version of the commit
message (to avoid a warning about ambiguous names) will be committed, with
the corresponding commit messages as contents.
If you need to provide a different file/different contents, you can use
the more explicit form
test_commit $MESSAGE $FILENAME $CONTENTS
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In 'man 1p trap' there is written:
"Implementations may permit names with the SIG prefix or ignore case
in signal names as an extension."
So change the lowercase signals to uppercase, which is POSIX compliant
instead of being an extension.
There wasn't anybody claiming that it doesn't work, but there was a bug
with using a signal with the SIG prefix, which is an extension as well.
So let's play it safe and change it, since it doesn't hurt anyone.
While at it, also convert 8 indentation spaces to 1 tab character.
Signed-off-by: Markus Heidelberg <markus.heidelberg@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In dc871831(Only use GIT_CONFIG in "git config", not other programs),
GIT_CONFIG_LOCAL was rested in peace, in favor of not reading
/etc/gitconfig and $HOME/.gitconfig at all when GIT_CONFIG is set.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* maint:
rebase -i: do not fail when there is no commit to cherry-pick
test-lib: fix color reset in say_color()
fix pread()'s short read in index-pack
Conflicts:
csum-file.c
When executing a single test with colors enabled, the cursor was not set
back to the previous one, and you had to hit an extra enter to get it
back.
Work around this problem by calling 'tput sgr0' before printing the
final newline.
Signed-off-by: Miklos Vajna <vmiklos@frugalware.org>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Sometimes you want to keep the trash directory, even if all tests
passed. For example, when extending tests, it comes it quite handy.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Many test scripts assumed that they will start in a 'trash' subdirectory
that is a single level down from the t/ directory, and referred to their
test vector files by asking for files like "../t9999/expect". This will
break if we move the 'trash' subdirectory elsewhere.
To solve this, we earlier introduced "$TEST_DIRECTORY" so that they can
refer to t/ directory reliably. This finally makes all the tests use
it to refer to the outside environment.
With this patch, and a one-liner not included here (because it would
contradict with what Dscho really wants to do):
| diff --git a/t/test-lib.sh b/t/test-lib.sh
| index 70ea7e0..60e69e4 100644
| --- a/t/test-lib.sh
| +++ b/t/test-lib.sh
| @@ -485,7 +485,7 @@ fi
| . ../GIT-BUILD-OPTIONS
|
| # Test repository
| -test="trash directory"
| +test="trash directory/another level/yet another"
| rm -fr "$test" || {
| trap - exit
| echo >&5 "FATAL: Cannot prepare test area"
all the tests still pass, but we would want extra sets of eyeballs on this
type of change to really make sure.
[jc: with help from Stephan Beyer on http-push tests I do not run myself;
credits for locating silly quoting errors go to Olivier Marin.]
Signed-off-by: Junio C Hamano <gitster@pobox.com>
On multiprocessor machines, or with I/O heavy tests (that leave the
CPU waiting a lot), it makes sense to parallelize the tests.
However, care has to be taken that the different jobs use different
trash directories.
This commit does so, by creating the trash directories with a suffix
that is unique with regard to the test, as it is the test's base name.
Further, the trash directory is removed in the test itself if
everything went fine, so that the trash directories do not
pile up only to be removed at the very end.
If a test failed, the trash directory is not removed. Chances are
that the exact error message is lost in the clutter, but you can still
see what test failed from the name of the trash directory, and repeat
the test (without -j).
If all was good, you will see the aggregated results.
Suggestions to simplify this commit came from Junio and René.
There still is an issue with tests that want to run a server process and
listen to a fixed port (http and svn) --- they cannot run in parallel but
this patch does not address this issue.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The test_must_fail function in test-lib.sh has been designed to
distinguish segmentation faults from controlled errors. But in the
current implementation this only works if a git command does not return a
small negative value, like -1, -2 or -3. But some git commands do.
Because any signal (like SIGSEGV) will result in an exit status
less than 193, this patch just adds a further check for the exit
status.
Signed-off-by: Stephan Beyer <s-beyer@gmx.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* dr/ceiling:
Eliminate an unnecessary chdir("..")
Add support for GIT_CEILING_DIRECTORIES
Fold test-absolute-path into test-path-utils
Implement normalize_absolute_path
Conflicts:
cache.h
setup.c
* sr/tests:
Hook up the result aggregation in the test makefile.
A simple script to parse the results from the testcases
Modify test-lib.sh to output stats to t/test-results/*
Conflicts:
t/test-lib.sh
This is for running external test scripts in other programming
languages that provide continuous output about their tests. Using
test_expect_success (like "test_expect_success 'description' 'perl
test-script.pl'") doesn't suffice here because test_expect_success
eats stdout in non-verbose mode, which is not fixable without major
file descriptor trickery.
Signed-off-by: Lea Wiemann <LeWiemann@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a --long-tests option to test-lib.sh, which enables tests to
selectively run more exhaustive (longer running, potentially
brute-force) tests. Such exhaustive tests would only be useful if one
works on the specific module that is being tested -- for a general "cd
t/; make" to check whether everything is OK, such exhaustive tests
shouldn't be run by default since the longer it takes to run the
tests, the less often they are actually run.
Signed-off-by: Lea Wiemann <LeWiemann@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Only ignore whitespace errors in t/tNNNN-*.sh and the t/tNNNN
subdirectories. Other files (like test libraries) should still be
checked.
Also fix a whitespace error in t/test-lib.sh.
Signed-off-by: Lea Wiemann <LeWiemann@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This change is needed order to aggregate data on the test run later on.
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Without this, some tests will fail because they compare command output
of subprocesses (such as git) with $PWD -- but subprocesses have the
physical path as their working directory, whereas $PWD contains the
symlinked path. This fixes it.
Signed-off-by: Lea Wiemann <LeWiemann@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Make git recognize a new environment variable that prevents it from
chdir'ing up into specified directories when looking for a GIT_DIR.
Useful for avoiding slow network directories.
For example, I use git in an environment where homedirs are automounted
and "ls /home/nonexistent" takes about 9 seconds. Setting
GIT_CEILING_DIRS="/home" allows "git help -a" (for bash completion) and
"git symbolic-ref" (for my shell prompt) to run in a reasonable time.
Signed-off-by: David Reiss <dreiss@facebook.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Exit with error if cd into the "trash directory" failed (error
already reported, so just exit).
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In order to help prevent regressions in the future, rename the trash directory
for all tests to contain spaces. This patch also corrects two failures that
were caused or exposed by this change.
Signed-off-by: Bryan Donlan <bdonlan@fushizen.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In particular, this function correctly handles cases where the pwd contains
spaces, quotes, and other troublesome metacharacters.
Signed-off-by: Bryan Donlan <bdonlan@fushizen.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When a git command is run under test_must_fail to make sure that
the argument parser catches bogus command line, it exits with 129.
We need to catch it as a valid "graceful error exit".
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When your test creates an unwritable directory that test framework cannot
clean out by "rm -fr trash", later tests cannot start in a fresh state
they expect to. Detect this and error out early.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Many scripts compare actual and expected output using
"diff -u". This is nicer than "cmp" because the output shows
how the two differ. However, not all versions of diff
understand -u, leading to unnecessary test failure.
This adds a test_cmp function to the test scripts and
switches all "diff -u" invocations to use it. The function
uses the contents of "$GIT_TEST_CMP" to compare its
arguments; the default is "diff -u".
On systems with a less-capable diff, you can do:
GIT_TEST_CMP=cmp make test
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* maint:
unquote_c_style: fix off-by-one.
test-lib: fix TERM to dumb for test repeatability
config.txt: refer to --upload-pack and --receive-pack instead of --exec
git-gui: Gracefully fall back to po2msg.sh if msgfmt --tcl fails
Dscho noticed that Term::ReadLine (used by send-email) colorized its
output for his TERM settings, inside t9001 tests.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we expect a git command to notice and signal errors, we
carelessly wrote in our tests:
test_expect_success 'reject bogus request' '
do something &&
do something else &&
! git command
'
but a non-zero exit could come from the "git command" segfaulting.
A new helper function "tset_must_fail" is introduced and it is
meant to be used to make sure the command gracefully fails (iow,
dying and exiting with non zero status is counted as a failure
to "gracefully fail"). The above example should be written as:
test_expect_success 'reject bogus request' '
do something &&
do something else &&
test_must_fail git command
'
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The top-level Makefile now creates a GIT-BUILD-OPTIONS file
which stores any options selected by the make process that
may be of use to further parts of the build process.
Specifically, we store the SHELL_PATH so that it can be used
by tests to construct shell scripts on the fly.
The format of the GIT-BUILD-OPTIONS file is Bourne shell,
and it is sourced by test-lib.sh; all tests can rely on just
having $SHELL_PATH correctly set in the environment.
The GIT-BUILD-OPTIONS file is written every time the
toplevel 'make' is invoked. Since the only users right now
are the test scripts, there's no drawback to updating its
timestamp. If something build-related depends on this, we
can do a trick similar to the one used by GIT-CFLAGS.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we expect a git command to notice and signal errors, we
carelessly wrote in our tests:
test_expect_success 'reject bogus request' '
do something &&
do something else &&
! git command
'
but a non-zero exit could come from the "git command" segfaulting.
A new helper function "tset_must_fail" is introduced and it is
meant to be used to make sure the command gracefully fails (iow,
dying and exiting with non zero status is counted as a failure
to "gracefully fail"). The above example should be written as:
test_expect_success 'reject bogus request' '
do something &&
do something else &&
test_must_fail git command
'
Signed-off-by: Junio C Hamano <gitster@pobox.com>
http-push tests require a web server with WebDAV support.
This commit introduces a HTTPD test library, which can be configured using
the following environment variables.
GIT_TEST_HTTPD enable HTTPD tests
LIB_HTTPD_PATH web server path
LIB_HTTPD_MODULE_PATH web server modules path
LIB_HTTPD_PORT listening port
LIB_HTTPD_DAV enable DAV
LIB_HTTPD_SVN enable SVN
LIB_HTTPD_SSL enable SSL
Signed-off-by: Clemens Buchacher <drizzd@aon.at>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Previously, we set the GIT_CONFIG environment variable in
our tests so that only that file was read. However, setting
it to a static value is not correct, since we are not
necessarily always in the same directory; instead, we want
the usual git config file lookup to happen.
To do this, we stop setting GIT_CONFIG, which means that we
must now suppress the reading of the system-wide and user
configs.
This exposes an incorrect test in t1500, which is also
fixed (the incorrect test worked because we were failing to
read the core.bare value from the config file, since the
GIT_CONFIG variable was pointing us to the wrong file).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we have known breakages, we still said "passed all N
test(s)", which was a bit funny.
This rewords it to read "passed all remaining N test(s)" in such
a case.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Originally, test_expect_failure was designed to be the opposite
of test_expect_success, but this was a bad decision. Most tests
run a series of commands that leads to the single command that
needs to be tested, like this:
test_expect_{success,failure} 'test title' '
setup1 &&
setup2 &&
setup3 &&
what is to be tested
'
And expecting a failure exit from the whole sequence misses the
point of writing tests. Your setup$N that are supposed to
succeed may have failed without even reaching what you are
trying to test. The only valid use of test_expect_failure is to
check a trivial single command that is expected to fail, which
is a minority in tests of Porcelain-ish commands.
This large-ish patch rewrites all uses of test_expect_failure to
use test_expect_success and rewrites the condition of what is
tested, like this:
test_expect_success 'test title' '
setup1 &&
setup2 &&
setup3 &&
! this command should fail
'
test_expect_failure is redefined to serve as a reminder that
that test *should* succeed but due to a known breakage in git it
currently does not pass. So if git-foo command should create a
file 'bar' but you discovered a bug that it doesn't, you can
write a test like this:
test_expect_failure 'git-foo should create bar' '
rm -f bar &&
git foo &&
test -f bar
'
This construct acts similar to test_expect_success, but instead
of reporting "ok/FAIL" like test_expect_success does, the
outcome is reported as "FIXED/still broken".
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This patch removes a spurious "command not found" error
and actually makes the "Test script did not set test_description."
string follow the command line option "--no-color".
Signed-off-by: Michele Ballabio <barra_cuda@katamail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This shuts down the "* ok ##: `test description`" messages.
Signed-off-by: Pierre Habouzit <madcoder@debian.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Johannes Schindelin
Junio C Hamano
Markus Heidelberg
Shawn O. Pearce
Miklos Vajna
Stephan Beyer
Lea Wiemann
Jakub Narebski
Sverre Rabbelier
David Reiss
Brandon Casey
Alex Riesen
Bryan Donlan
Jeff King
Clemens Buchacher
Michele Ballabio
Pierre Habouzit