The ctypes module is used on windows to calculate free disk space,
so it must be imported. We won't need it on other platforms, but
the module is available in Python 2.5 and newer, so importing it
unconditionally is harmless.
Signed-off-by: Dennis Kaarsemaker <dennis@kaarsemaker.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Teach "git p4" to send large blobs outside the repository by
talking to Git LFS.
* ls/p4-lfs:
git-p4: add Git LFS backend for large file system
git-p4: add support for large file systems
git-p4: check free space during streaming
git-p4: add file streaming progress in verbose mode
git-p4: return an empty list if a list config has no values
git-p4: add gitConfigInt reader
git-p4: add optional type specifier to gitConfig reader
Work around "git p4" failing when the P4 depot records the contents
in UTF-16 without UTF-16 BOM.
* ls/p4-translation-failure:
git-p4: handle "Translation of file content failed"
git-p4: add test case for "Translation of file content failed" error
"git p4" learned to reencode the pathname it uses to communicate
with the p4 depot with a new option.
* ls/p4-path-encoding:
git-p4: use replacement character for non UTF-8 characters in paths
git-p4: improve path encoding verbose output
git-p4: add config git-p4.pathEncoding
Correct "git p4 --detect-labels" so that it does not fail to create
a tag that points at a commit that is also being imported.
* ld/p4-import-labels:
git-p4: fix P4 label import for unprocessed commits
git-p4: do not terminate creating tag for unknown commit
git-p4: failing test for ignoring invalid p4 labels
Add example implementation including test cases for the large file
system using Git LFS.
Pushing files to the Git LFS server is not tested.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Perforce repositories can contain large (binary) files. Migrating these
repositories to Git generates very large local clones. External storage
systems such as Git LFS [1], Git Fat [2], Git Media [3], git-annex [4]
try to address this problem.
Add a generic mechanism to detect large files based on extension,
uncompressed size, and/or compressed size.
[1] https://git-lfs.github.com/
[2] https://github.com/jedbrown/git-fat
[3] https://github.com/alebedev/git-media
[4] https://git-annex.branchable.com/
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Conflicts:
Documentation/git-p4.txt
git-p4.py
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-p4 will just halt if there is not enough disk space while
streaming content from P4 to Git. Add a check to ensure at least
4 times (arbitrarily chosen) the size of a streamed file is available.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If a file is streamed from P4 to Git then the verbose mode prints
continuously the progress as percentage like this:
//depot/file.bin 20% (10 MB)
Upon completion the progress is overwritten with depot source, local
file and size like this:
//depot/file.bin --> local/file.bin (10 MB)
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a git config reader for integer variables. Please note that the
git config implementation automatically supports k, m, and g suffixes.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The functions "gitConfig" and "gitConfigBool" are almost identical.
Make "gitConfig" more generic by adding an optional type specifier.
Use the type specifier "--bool" with "gitConfig" to implement
"gitConfigBool. This prepares the implementation of other type
specifiers such as "--int".
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A P4 repository can get into a state where it contains a file with
type UTF-16 that does not contain a valid UTF-16 BOM. If git-p4
attempts to retrieve the file then the process crashes with a
"Translation of file content failed" error.
More info here: http://answers.perforce.com/articles/KB/3117
Fix this by detecting this error and retrieving the file as binary
instead. The result in Git is the same.
Known issue: This works only if git-p4 is executed in verbose mode.
In normal mode no exceptions are thrown and git-p4 just exits.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If non UTF-8 characters are detected in paths then replace them with
a placeholder instead of throwing a UnicodeDecodeError
exception. This restores the original (implicit) implementation that
was broken in 00a9403.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Reviewed-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If a path with non-ASCII characters is detected then print the
encoding and the encoded string in verbose mode.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Perforce keeps the encoding of a path as given by the originating OS.
Git expects paths encoded as UTF-8. Add a config to tell git-p4 what
encoding Perforce had used for the paths. This encoding is used to
transcode the paths to UTF-8. As an example, Perforce on Windows often
uses “cp1252” to encode path names.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
With --detect-labels enabled, git-p4 will try to create tags
using git fast-import by writing a "tag" clause to the
fast-import stream.
If the commit that the tag references has not yet actually
been processed by fast-import, then the tag can't be created
and git-p4 fails to import the P4 label.
Teach git-p4 to use fast-import "marks" when creating tags
which reference commits created during the current run of the
program.
Commits created before the current run are still referenced
in the old way using a normal git commit.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If p4 reports a tag for a commit that git-p4 does not know
about (e.g. because it references a P4 changelist that was
imported prior to the point at which the repo was cloned into
git), make sure that the error is correctly caught and handled.
rather than just crashing.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Perforce depot may record paths in mixed cases, e.g. "p4 files" may
show that there are these two paths:
//depot/Path/to/file1
//depot/pATH/to/file2
and with "p4" or "p4v", these end up in the same directory, e.g.
//depot/Path/to/file1
//depot/Path/to/file2
which is the desired outcome on case insensitive systems.
If git-p4 is used with client spec "//depot/Path/...", however, then
all files not matching the case in the client spec are ignored (in
the example above "//depot/pATH/to/file2").
Fix this by using the path case that appears first in lexicographical
order when core.ignorecase is set to true. This behavior is consistent
with "p4" and "p4v".
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
More Perforce row number limit workaround for "git p4".
* ld/p4-changes-block-size:
git-p4: fixing --changes-block-size handling
git-p4: add tests for non-numeric revision range
git-p4: test with limited p4 server results
git-p4: additional testing of --changes-block-size
The --changes-block-size handling was intended to help when
a user has a limited "maxscanrows" (see "p4 group"). It used
"p4 changes -m $maxchanges" to limit the number of results.
Unfortunately, it turns out that the "maxscanrows" and "maxresults"
limits are actually applied *before* the "-m maxchanges" parameter
is considered (experimentally).
Fix the block-size handling so that it gets blocks of changes
limited by revision number ($Start..$Start+$N, etc). This limits
the number of results early enough that both sets of tests pass.
Note that many other Perforce operations can fail for the same
reason (p4 print, p4 files, etc) and it's probably not possible
to workaround this. In the real world, this is probably not
usually a problem.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Unlike "$EDITOR" and "$GIT_EDITOR" that can hold the path to the
command and initial options (e.g. "/path/to/emacs -nw"), 'git p4'
did not let the shell interpolate the contents of the environment
variable that name the editor "$P4EDITOR" (and "$EDITOR", too).
Make it in line with the rest of Git, as well as with Perforce.
* ld/p4-editor-multi-words:
git-p4: tests: use test-chmtime in place of touch
git-p4: fix handling of multi-word P4EDITOR
git-p4: add failing test for P4EDITOR handling
Fixing bug with UTF-16 files when they are retrieved by git-p4. It
was always getting the tip version of the file and the history of
the file was lost.
Signed-off-by: Miguel Torroja <miguel.torroja@gmail.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This teaches git-p4 to pass the P4EDITOR variable to the
shell for expansion, so that any command-line arguments are
correctly handled. Without this, git-p4 can only launch the
editor if P4EDITOR is solely the path to the binary, without
any arguments.
This also adjusts t9805, which relied on the previous behaviour.
Suggested-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git p4 attempts to better handle branches in Perforce.
* va/p4-client-path:
git-p4: improve client path detection when branches are used
t9801: check git-p4's branch detection with client spec enabled
"git p4" learned "--changes-block-size <n>" to read the changes in
chunks from Perforce, instead of making one call to "p4 changes"
that may trigger "too many rows scanned" error from Perforce.
* ls/p4-changes-block-size:
git-p4: use -m when running p4 changes
Perforce allows client side file/directory remapping through
the use of the client view definition that is part of the
user's client spec.
To support this functionality while branch detection is
enabled it is important to determine the branch location in
the workspace such that the correct files are patched before
Perforce submission. Perforce provides a command that
facilitates this process: p4 where.
This patch does two things to fix improve file location
detection when git-p4 has branch detection and use of client
spec enabled:
1. Enable usage of "p4 where" when Perforce branches exist
in the git repository, even when client specification is
used. This makes use of the already existing function
p4Where.
2. Allow identifying partial matches of the branch's depot
path while processing the output of "p4 where". For
robustness, paths will only match if ending in "/...".
Signed-off-by: Vitor Antunes <vitor.hda@gmail.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Simply running "p4 changes" on a large branch can result in a "too
many rows scanned" error from the Perforce server. It is better to
use a sequence of smaller calls to "p4 changes", using the "-m"
option to limit the size of each call.
Signed-off-by: Lex Spoon <lex@lexspoon.org>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If a Perforce server is configured to automatically set +l
(exclusive lock) on add of certain file types, git p4 submit will
fail during getP4OpenedType, as the regex doesn't expect the
trailing '*exclusive*' from p4 opened:
//depot/file.png#1 - add default change (binary+l) *exclusive*
Signed-off-by: Blair Holloway <blair_holloway@playstation.sony.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The clone subcommand has long had support for excluding
subdirectories, but sync has not. This is a nuisance,
since as soon as you do a sync, any changed files that
were initially excluded start showing up.
Move the "exclude" command-line option into the parent
class; the actual behavior was already present there so
it simply had to be exposed.
Signed-off-by: Luke Diamand <luke@diamand.org>
Reviewed-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If you use git-p4 with the "--prepare-p4-only" option, then
it prints the p4 command line to use. However, the command
line was incorrect: the changelist specification must be
supplied on standard input, not as an argument to p4.
Signed-off-by: Luke Diamand <luke@diamand.org>
Acked-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
b4073bb3 (git-p4: Do not include diff in spec file when just
preparing p4, 2014-05-24) broke git p4 submit, here is a proper
fix, including proper handling for windows end of lines.
Signed-off-by: Maxime Coste <frrrwww@gmail.com>
Acked-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The diff information render the spec file unusable as is by p4,
do not include it when run with --prepare-p4-only so that the
given file can be directly passed to p4.
With --prepare-p4-only, git-p4 already tells the user it can use
p4 submit with the generated spec file. This fails because of the
diff being present in the file. Not including the diff fixes that.
Without --prepare-p4-only, keeping the diff makes sense for a
quick review of the patch before submitting it. And does not cause
problems with p4 as we remove it programmatically.
Signed-off-by: Maxime Coste <frrrwww@gmail.com>
Acked-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Fixes a regression in 1.9.0 with an obviously correct single-liner.
* cl/p4-use-diff-tree:
git-p4: format-patch to diff-tree change breaks binary patches
When applying binary patches a full index is required. format-patch
already handles this, but diff-tree needs '--full-index' argument
to always output full index. When git-p4 runs git-apply to test
the patch, git-apply rejects the patch due to abbreviated blob
object names. This is the error message git-apply emits in this
case:
error: cannot apply binary patch to '<filename>' without full index line
error: <filename>: patch does not apply
Signed-off-by: Tolga Ceylan <tolga.ceylan@gmail.com>
Acked-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
'git p4 rebase' fails with the following message if there is a file
named HEAD in the current directory:
fatal: ambiguous argument 'HEAD': both revision and filename
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Take the suggestion above and explicitly state that HEAD should be
treated as a revision.
Signed-off-by: Vlad Dogaru <vdogaru@ixiacom.com>
Acked-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When "p4 where" fails, for whatever reason, the error message tries to
show an undefined variable. This minor bug applies only when using a
client spec, and was introduced recently in 9d57c4a (git p4: implement
view spec wildcards with "p4 where", 2013-08-30).
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Commit 9d7d446 (git p4: submit files with wildcards, 2012-04-29)
fixed problems with handling files that had p4 wildcard
characters, like "@" and "*". But it missed one case, that of
RCS keyword scrubbing, which uses "p4 fstat" to extract type
information. Fix it by calling wildcard_encode() on the raw
filename.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Generating the submit template for p4 uses tempfile.mkstemp(),
which by default puts files in /tmp. For a test that fails,
possibly on purpose, this is not cleaned up. Run with TMPDIR
pointing into the trash directory so the temp files go away
with the test results.
To do this required some other minor changes. First, the editor
is launched using system(editor + " " + template_file), using
shell expansion to build the command string. This doesn't work
if editor has a space in it. And is generally unwise as it's
easy to fool the shell into doing extra work. Exec the args
directly, without shell expansion.
Second, without shell expansion, the trick of "P4EDITOR=:" used
in the tests doesn't work. Use a real command, true, as the
non-interactive editor for testing.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Damien Gérard highlights an interesting problem. Some p4
repositories end up with symlinks that have an empty target. It
is not possible to create this with current p4, but they do
indeed exist.
The effect in git p4 is that "p4 print" on the symlink returns an
empty string, confusing the curret symlink-handling code.
Such broken repositories cause problems in p4 as well, even with
no git involved. In p4, syncing to a change that includes a
bogus symlink causes errors:
//depot/empty-symlink - updating /home/me/p4/empty-symlink
rename: /home/me/p4/empty-symlink: No such file or directory
and leaves no symlink.
In git, replicate the p4 behavior by ignoring these bad symlinks.
If, in a later p4 revision, the symlink happens to point to
something non-null, the symlink will be replaced properly.
Add a big test for all this too.
This happens to be a regression introduced by 1292df1 (git-p4:
Fix occasional truncation of symlink contents., 2013-08-08) and
appeared first in 1.8.5. But it shows up only in p4 repositories
of dubious character, so can wait for a proper release.
Tested-by: Damien Gérard <damien@iwi.me>
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The output of git format-patch can vary with user preferences. In
particular setting diff.noprefix will break the "git apply" that
is done as part of "git p4 submit".
Acked-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Crestez Dan Leonard <cdleonard@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git p4" does not support many of the view wildcards, such as * and
%%n. It only knows the common ... mapping, and exclusions.
Redo the entire wildcard code around the idea of directly querying
the p4 server for the mapping. For each commit, invoke "p4 where"
with committed file paths as args and use the client mapping to
decide where the file goes in git.
This simplifies a lot of code, and adds support for all wildcards
supported by p4. Downside is that there is probably a 20%-ish
slowdown with this approach.
[pw: redo code and tests]
Signed-off-by: Kazuki Saitoh <ksaitoh560@gmail.com>
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Symlink contents in p4 print sometimes have a trailing
new line character, but sometimes it doesn't. git-p4
should only remove the last character if that character
is '\n'.
Signed-off-by: Alex Juncu <ajuncu@ixiacom.com>
Signed-off-by: Alex Badea <abadea@ixiacom.com>
Acked-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Junio C Hamano
Dennis Kaarsemaker
Lars Schneider
Luke Diamand
Miguel Torroja
Vitor Antunes
Lex Spoon
Blair Holloway
Maxime Coste
Tolga Ceylan
Vlad Dogaru
Pete Wyckoff
Crestez Dan Leonard
Kazuki Saitoh
Alexandru Juncu
Ondřej Bílka