"git filter-branch" in a repository with many refs blew limit of
command line length.
* lc/filter-branch-too-many-refs:
Allow git-filter-branch to process large repositories with lots of branches.
A recommended way to move trees between repositories is to use
git-filter-branch to revise the history for a single tree:
However, this can lead to "argument list too long" errors when the
original repository has many retained branches (>6k)
/usr/local/git/libexec/git-core/git-filter-branch: line 270:
/usr/local/git/libexec/git-core/git: Argument list too long
Could not get the commits
Saving the output from rev-parse and feeding it into rev-list from
its standard input avoids this problem, since the rev-parse output
is not processed as a command line argument.
Signed-off-by: Lee Carver <Lee.Carver@servicenow.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Commit 4337b58 (do not write null sha1s to on-disk index,
2012-07-28) added a safety check preventing git from writing
null sha1s into the index. The intent was to catch errors in
other parts of the code that might let such an entry slip
into the index (or worse, a tree).
Some existing repositories may have invalid trees that
contain null sha1s already, though. Until 4337b58, a common
way to clean this up would be to use git-filter-branch's
index-filter to repair such broken entries. That now fails
when filter-branch tries to write out the index.
Introduce a GIT_ALLOW_NULL_SHA1 environment variable to
relax this check and make it easier to recover from such a
history.
It is tempting to not involve filter-branch in this commit
at all, and instead require the user to manually invoke
GIT_ALLOW_NULL_SHA1=1 git filter-branch ...
to perform an index-filter on a history with trees with null
sha1s. That would be slightly safer, but requires some
specialized knowledge from the user. So let's set the
GIT_ALLOW_NULL_SHA1 variable automatically when checking out
the to-be-filtered trees. Advice on using filter-branch to
remove such entries already exists on places like
stackoverflow, and this patch makes it Just Work again on
recent versions of git.
Further commands that touch the index will still notice and
fail, unless they actually remove the broken entries. A
filter-branch whose filters do not touch the index at all
will not error out (since we complain of the null sha1 only
on writing, not when making a tree out of the index), but
this is acceptable, as we still print a loud warning, so the
problem is unlikely to go unnoticed.
Signed-off-by: Jeff King <peff@peff.net>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When used with "-d temporary-directory" option, "git filter-branch"
failed to come back to the original working tree to perform the
final clean-up procedure.
* jk/filter-branch-come-back-to-original:
filter-branch: return to original dir after filtering
The first thing filter-branch does is to create a temporary
directory, either ".git-rewrite" in the current directory
(which may be the working tree or the repository if bare),
or in a directory specified by "-d". We then chdir to
$tempdir/t as our temporary working directory in which to run
tree filters.
After finishing the filter, we then attempt to go back to
the original directory with "cd ../..". This works in the
.git-rewrite case, but if "-d" is used, we end up in a
random directory. The only thing we do after this chdir is
to run git-read-tree, but that means that:
1. The working directory is not updated to reflect the
filtered history.
2. We dump random files into "$tempdir/.." (e.g., if you
use "-d /tmp/foo", we dump junk into /tmp).
Fix it by recording the full path to the original directory
and returning there explicitly.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This saves us some code, but it also reduces the number of
processes we start for each filtered commit. Since we can
parse both author and committer in the same sed invocation,
we save one process. And since the new interface avoids tr,
we save 4 processes.
It also avoids using "tr", which has had some odd
portability problems reported with from Solaris's xpg6
version.
We also tweak one of the tests in t7003 to double-check that
we are properly exporting the variables (because test-lib.sh
exports GIT_AUTHOR_NAME, it will be automatically exported
in subprograms. We override this to make sure that
filter-branch handles it properly itself).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In 1.7.9 era, we taught "git rebase" about the raw timestamp format
but we did not teach the same trick to "filter-branch", which rolled
a similar logic on its own. Because of this, "filter-branch" failed
to rewrite commits with ancient timestamps.
* jc/maint-filter-branch-epoch-date:
t7003: add test to filter a branch with a commit at epoch
date.c: Fix off by one error in object-header date parsing
filter-branch: do not forget the '@' prefix to force git-timestamp
For some reason, this script reinvents, instead of refactoring the
existing one in git-sh-setup, the logic to grab ident information
from an existing commit; it was missed when the corresponding logic
in git-sh-setup was updated with 2c733fb (parse_date(): '@' prefix
forces git-timestamp, 2012-02-02).
Teach the script that it is OK to have a way ancient timestamp in
the commits that are being filtered.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Filter-branch already requires that we have a clean work
tree before starting. However, it failed to refresh the
index before checking, which means it could be wrong in the
case of stat-dirtiness.
Instead of simply adding a call to refresh the index, let's
switch to using the require_clean_work_tree function
provided by git-sh-setup. It does exactly what we want, and
with fewer lines of code and more specific output messages.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
According to `git help filter-branch':
--commit-filter <command>
...
You can use the _map_ convenience function in this filter,
and other convenience functions, too...
...
However, it turns out that `map' hasn't been usable because it depends
on the variable `workdir', which is not propogated to the environment
of the shell that runs the commit-filter <command> because the
shell is created via a simple-command rather than a compound-command
subshell:
@SHELL_PATH@ -c "$filter_commit" "git commit-tree" \
$(git write-tree) $parentstr < ../message > ../map/$commit ||
die "could not write rewritten commit"
One solution is simply to export `workdir'. However, it seems rather
heavy-handed to export `workdir' to the environments of all commands,
so instead this commit exports `workdir' for only the duration of the
shell command in question:
workdir=$workdir @SHELL_PATH@ -c "$filter_commit" "git commit-tree" \
$(git write-tree) $parentstr < ../message > ../map/$commit ||
die "could not write rewritten commit"
Signed-off-by: Michael Witten <mfwitten@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"Usage: git filter-branch " that is prefixed to the first line is 25
columns long, so the "[--index-filter ..." on the second line would not
align with "[--env-filter ..." on the first line to begin with. If the
second and subsequent lines do not aim to align with anything on the
first line, it is just fine to indent them with a single HT.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
I found that the patched 4 files were different when this
filter is applied.
expand -i | unexpand --first-only
This patch contains the corrected files.
Signed-off-by: Jon Seymour <jon.seymour@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We can be clever and know by ourselves when we need the behavior
implied by "--remap-to-ancestor". No need to encumber users by having
them exposed to it as a tunable. (Option kept for backward compatibility,
but it's now a no-op.)
Signed-off-by: Csaba Henk <csaba@gluster.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Running filter-branch with --prune-empty and --commit-filter reports:
"Cannot set --prune-empty and --filter-commit at the same time".
Change it to use the correct option name: --commit-filter
Signed-off-by: Jacob Helwig <jacob.helwig@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* ms/filter-branch-submodule:
filter-branch: Add tests for submodules in tree-filter
filter-branch: Fix to allow replacing submodules with another content
When git filter-branch is used to replace a submodule with another
content, it always fails on the first commit.
Consider a repository with submod directory containing a submodule. The
following command to remove the submodule and replace it with a file fails:
git filter-branch --tree-filter 'rm -rf submod &&
git rm -q submod &&
mkdir submod &&
touch submod/file'
with an error:
error: submod: is a directory - add files inside instead
The reason is that git diff-index, which generates the first part of the
list of files updated by the tree filter, emits also the removed submodule
even if it was replaced by a real directory.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Using a dollar sign in double quotes isn't portable. Escape them with
a backslash or replace the double quotes with single quotes.
Signed-off-by: Stephen Boyd <bebarino@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The intent of this particular call to 'git read-tree' was to fill an
index. But in fact, it only allocated an empty index. Later in the
program, the index is filled anyway by calling read-tree with specific
commits, and considering that elsewhere the index is even removed (i.e.,
it is not relied upon that the index file exists), this first call of
read-tree is completely redundant.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If the user has exported the GREP_OPTIONS environment variable, the output
from "grep" and "egrep" in scripted Porcelains may be different from what
they expect. For example, we may want to count number of matching lines,
by "grep" piped to "wc -l", and GREP_OPTIONS=-C3 will break such use.
The approach taken by this change to address this issue is to protect only
our own use of grep/egrep. Because we do not unset it at the beginning of
our scripts, hook scripts run from the scripted Porcelains are exposed to
the same insanity this environment variable causes when grep/egrep is used
to implement logic (e.g. "grep | wc -l"), and it is entirely up to the
hook scripts to protect themselves.
On the other hand, applypatch-msg hook may want to show offending words in
the proposed commit log message using grep to the end user, and the user
might want to set GREP_OPTIONS=--color to paint the match more visibly.
The approach to protect only our own use without unsetting the environment
variable globally will allow this use case.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since a0e4639 (filter-branch: fix ref rewriting with
--subdirectory-filter, 2008-08-12) git-filter-branch has done
nearest-ancestor rewriting when using a --subdirectory-filter.
However, that rewriting strategy is also a useful building block in
other tasks. For example, if you want to split out a subset of files
from your history, you would typically call
git filter-branch -- <refs> -- <files>
But this fails for all refs that do not point directly to a commit
that affects <files>, because their referenced commit will not be
rewritten and the ref remains untouched.
The code was already there for the --subdirectory-filter case, so just
introduce an option that enables it independently.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Handling $filter_subdir in the usual way requires a separate case at
every use, because the variable is empty when unused.
Furthermore, --subdirectory-filter supplies its own '--', and if the user
provided one himself, such as in
git filter-branch --subdirectory-filter subdir -- --all -- subdir/file
an extra '--' was used as path filter in the call to git-rev-list that
determines the commits that shall be rewritten.
To keep the argument handling sane, we filter $@ to contain only the
non-revision arguments, and store all revisions in $ref_args. The
$ref_args are easy to handle since only the SHA1s are needed; the
actual branch names have already been stored in $tempdir/heads at this
point.
An extra separating -- is only required if the user did not provide
any non-revision arguments, as the latter disambiguate the
$filter_subdir following after them (or fail earlier because they are
ambiguous themselves).
Thanks to Johannes Sixt for suggesting this solution.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
It used to be a single, huge line, badly wrapped by xterm.
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A minor fix that eliminates usage of "2>/dev/null" when --quiet or
-q has already been implemented.
Signed-off-by: Dan Loewenherz <daniel.loewenherz@yale.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Avoid using simple variable names like 'i', since user commands are eval'ed
and may clash with and overwrite our values.
Signed-off-by: Elijah Newren <newren@gmail.com>
Acked-by: Petr Baudis <pasky@suse.cz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Tell the user that a backup (original) already exists, and how to solve
this problem (with -f option)
Signed-off-by: John Tapsell <johnflux@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The improved error handling catches a bug in filter-branch when using
-d pointing to a path outside any git repository:
$ git filter-branch -d /tmp/foo master
fatal: Not a git repository (or any of the parent directories): .git
This error message comes from git for-each-ref in line 224. GIT_DIR is
set correctly by git-sh-setup (to the foo.git repository), but not
exported (yet).
Signed-off-by: Lars Noschinski <lars@public.noschinski.de>
Acked-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
9273b56 (filter-branch: Fix fatal error on bare repositories, 2009-02-03)
fixed a missing check of return status from an underlying command in
git-filter-branch, but there still are places that do not check errors.
For example, the command does not pay attention to the exit status of the
command given by --commit-filter. It should abort in such a case.
This attempts to fix all the remaining places that fails to checks errors.
In two places, I've had to break apart pipelines in order to check the
error code for the first stage of the pipeline, as discussed here:
http://kerneltrap.org/mailarchive/git/2009/1/28/4835614
Feedback on this patch was provided by Johannes Sixt, Johannes Schindelin
and Junio C Hamano. Thomas Rast helped with pipeline error handling.
Signed-off-by: Eric Kidd <git@randomhacks.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* js/filter-branch-submodule:
filter-branch: do not consider diverging submodules a 'dirty worktree'
filter-branch: Fix fatal error on bare repositories
At the end of filter-branch in a non-bare repository, the work tree is
updated with "read-tree -m -u HEAD", to carry the change forward in case
the current branch was rewritten. In order to avoid losing any local
change during this step, filter-branch refuses to work when there are
local changes in the work tree.
This "read-tree -m -u HEAD" operation does not affect what commit is
checked out in a submodule (iow, it does not touch .git/HEAD in a
submodule checkout), and checking if there is any local change to the
submodule is not useful.
Staged submodules _are_ considered to be 'dirty', however, as the
"read-tree -m -u HEAD" could result in loss of staged information
otherwise.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When git filter-branch is run on a bare repository, it prints out a fatal
error message:
$ git filter-branch branch
Rewrite 476c4839280c219c2317376b661d9d95c1727fc3 (9/9)
WARNING: Ref 'refs/heads/branch' is unchanged
fatal: This operation must be run in a work tree
Note that this fatal error message doesn't prevent git filter-branch from
exiting successfully. (Why doesn't git filter-branch actually exit with an
error when a shell command fails? I'm not sure why it was designed this
way.)
This error message is caused by the following section of code at the end of
git-filter-branch.sh:
if [ "$(is_bare_repository)" = false ]; then
unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE
test -z "$ORIG_GIT_DIR" || {
GIT_DIR="$ORIG_GIT_DIR" && export GIT_DIR
}
... elided ...
git read-tree -u -m HEAD
fi
The problem is the call to $(is_bare_repository), which is made before
GIT_DIR and GIT_WORK_TREE are restored. This call always returns "false",
even when we're running in a bare repository. But this means that we will
attempt to call 'git read-tree' even in a bare repository, which will fail
and print an error.
This patch modifies git-filter-branch.sh to restore the original
environment variables before trying to call is_bare_repository.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git_commit_non_empty_tree is added to the functions that can be run from
commit filters. Its effect is to commit only commits actually touching the
tree and that are not merge points either.
The option --prune-empty is added. It defaults the commit-filter to
'git_commit_non_empty_tree "$@"', and can be used with any other
combination of filters, except --commit-hook that must used
'git_commit_non_empty_tree "$@"' where one puts 'git commit-tree "$@"'
usually to achieve the same result.
Signed-off-by: Pierre Habouzit <madcoder@debian.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* tr/filter-branch:
revision --simplify-merges: make it a no-op without pathspec
revision --simplify-merges: do not leave commits unprocessed
revision --simplify-merges: use decoration instead of commit->util field
Documentation: rev-list-options: move --simplify-merges documentation
filter-branch: use --simplify-merges
filter-branch: fix ref rewriting with --subdirectory-filter
filter-branch: Extend test to show rewriting bug
Topo-sort before --simplify-merges
revision traversal: show full history with merge simplification
revision.c: whitespace fix
The tag rewriting code used a 'sed' expression to substitute the new tag
name into the corresponding field of the annotated tag object. But this is
problematic if the tag name contains special characters. In particular,
if the tag name contained a slash, then the 'sed' expression had a syntax
error. We now protect against this by using 'printf' to assemble the
tag header.
Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Use rev-list --simplify-merges everywhere. This changes the behaviour
of --subdirectory-filter in cases such as
O -- A -\
\ \
\- B -- M
where A and B bring the same changes to the subdirectory: It now keeps
both sides of the merge. Previously, the history would have been
simplified to 'O -- A'. Merges of unrelated side histories that never
touch the subdirectory are still removed.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The previous ancestor discovery code failed on any refs that are
(pre-rewrite) ancestors of commits marked for rewriting. This means
that in a situation
A -- B(topic) -- C(master)
where B is dropped by --subdirectory-filter pruning, the 'topic' was
not moved up to A as intended, but left unrewritten because we asked
about 'git rev-list ^master topic', which does not return anything.
Instead, we use the straightforward
git rev-list -1 $ref -- $filter_subdir
to find the right ancestor. To justify this, note that the nearest
ancestor is unique: We use the output of
git rev-list --parents -- $filter_subdir
to rewrite commits in the first pass, before any ref rewriting. If B
is a non-merge commit, the only candidate is its parent. If it is a
merge, there are two cases:
- All sides of the merge bring the same subdirectory contents. Then
rev-list already pruned away the merge in favour for just one of its
parents, so there is only one candidate.
- Some merge sides, or the merge outcome, differ. Then the merge is
not pruned and can be rewritten directly.
So it is always safe to use rev-list -1.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Previously, git-filter-branch failed if it attempted to update an
annotated tag. Now we ignore this condition if --tag-name-filter is
given, so that we can later rewrite the tag. If no such option was
provided, we warn the user that he might want to run with
"--tag-name-filter cat" to achieve the intended effect.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Commit 46eb449c restricted git-filter-branch to non-bare repositories
unnecessarily; git-filter-branch can work on bare repositories just
fine.
Cc: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Petr Baudis <pasky@suse.cz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This reverts commit cfabd6eee1. I had
implemented it without understanding what --full-history does. Consider
this history:
C--M--N
/ / /
A--B /
\ /
D-/
where B and C modify a path, X, in the same way so that the result is
identical, and D does not modify it at all. With the path limiter X and
without --full-history this is simplified to
A--B
i.e. only one of the paths via B or C is chosen. I had assumed that
--full-history would keep both paths like this
C--M
/ /
A--B
removing the path via D; but in fact it keeps the entire history.
Currently, git does not have the capability to simplify to this
intermediary case. However, the other extreme to keep the entire history
is not wanted either in usual cases. I think we can expect that histories
like the above are rare, and in the usual cases we want a simplified
history. So let's remove --full-history again.
(Concerning t7003, subsequent tests depend on what the test case sets up,
so we can't just back out the entire test case.)
Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
filter-branch tries to restore "old" copies of some
environment variables by using the construct:
unset var
test -z "$old_var" || var="$old_var" && export var
This is just wrong. AND-list and OR-list operators && and || have equal
precedence and they bind left to right. The second term, var="$old"
assignment always succeeds, so we always end up exporting var.
On bash and dash, exporting an unset variable has no effect. However, on
some shells (such as FreeBSD's /bin/sh), the shell exports the empty
value.
This manifested itself in this case as git-filter-branch setting
GIT_INDEX_FILE to the empty string, which in turn caused its call to
git-read-tree to fail, leaving the working tree pointing at the original
HEAD instead of the rewritten one.
To fix this, we change the short-circuit logic to better match the intent:
test -z "$old_var" || {
var="$old_var" && export var
}
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Commit d89c1df (filter-branch: don't use xargs -0, 2008-03-12) replaced a
'ls-files | xargs rm' pipeline by 'git clean'. 'git clean' however does
not recurse and remove directories by default.
Now, consider a tree-filter that renames a directory.
1. For the first commit everything works as expected
2. Then filter-branch checks out the files for the next commit. This
leaves the new directory behind because there is no real "branch
switching" involved that would notice that the directory can be
removed.
3. Then filter-branch invokes 'git clean' to remove exactly those
left-overs. But here it does not remove the directory.
4. The next tree-filter does not work as expected because there already
exists a directory with the new name.
Just add -d to 'git clean', so that empty directories are removed.
Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add support for creating a new tag object and retaining the tag message,
author, and date when rewriting tags. The gpg signature, if one exists,
will be stripped.
This adds nearly proper tag name filtering to filter-branch. Proper tag
name filtering would include the ability to change the tagger, tag date,
tag message, and _not_ strip a gpg signature if the tag did not change.
Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
On some systems, 'sh' isn't very friendly. In particular,
t7003 fails on Solaris because it doesn't understand $().
Instead, use the specified SHELL_PATH to run shell code.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Junio C Hamano
Lee Carver
Jeff King
Michael Witten
Jon Seymour
Csaba Henk
Jacob Helwig
Michal Sojka
Stephen Boyd
Johannes Sixt
Thomas Rast
Matthieu Moy
Dan Loewenherz
Elijah Newren
John Tapsell
Lars Noschinski
Eric Kidd
Johannes Schindelin
Pierre Habouzit
Miklos Vajna
Johannes Sixt
Petr Baudis
veillette@yahoo.ca
Brandon Casey