Merge branch 'ls/filter-process'

The smudge/clean filter API expect an external process is spawned
to filter the contents for each path that has a filter defined.  A
new type of "process" filter API has been added to allow the first
request to run the filter for a path to spawn a single process, and
all filtering need is served by this single process for multiple
paths, reducing the process creation overhead.

* ls/filter-process:
  contrib/long-running-filter: add long running filter example
  convert: add filter.<driver>.process option
  convert: prepare filter.<driver>.process option
  convert: make apply_filter() adhere to standard Git error handling
  pkt-line: add functions to read/write flush terminated packet streams
  pkt-line: add packet_write_gently()
  pkt-line: add packet_flush_gently()
  pkt-line: add packet_write_fmt_gently()
  pkt-line: extract set_packet_header()
  pkt-line: rename packet_write() to packet_write_fmt()
  run-command: add clean_on_exit_handler
  run-command: move check_pipe() from write_or_die to run_command
  convert: modernize tests
  convert: quote filter names in error messages
This commit is contained in:
Junio C Hamano 2016-10-31 13:15:21 -07:00
Родитель 906d6906fb 0f71fa273f
Коммит dbaa6bdce2
19 изменённых файлов: 1498 добавлений и 135 удалений

Просмотреть файл

@ -293,7 +293,15 @@ checkout, when the `smudge` command is specified, the command is
fed the blob object from its standard input, and its standard
output is used to update the worktree file. Similarly, the
`clean` command is used to convert the contents of worktree file
upon checkin.
upon checkin. By default these commands process only a single
blob and terminate. If a long running `process` filter is used
in place of `clean` and/or `smudge` filters, then Git can process
all blobs with a single filter command invocation for the entire
life of a single Git command, for example `git add --all`. If a
long running `process` filter is configured then it always takes
precedence over a configured single blob filter. See section
below for the description of the protocol used to communicate with
a `process` filter.
One use of the content filtering is to massage the content into a shape
that is more convenient for the platform, filesystem, and the user to use.
@ -373,6 +381,153 @@ not exist, or may have different contents. So, smudge and clean commands
should not try to access the file on disk, but only act as filters on the
content provided to them on standard input.
Long Running Filter Process
^^^^^^^^^^^^^^^^^^^^^^^^^^^
If the filter command (a string value) is defined via
`filter.<driver>.process` then Git can process all blobs with a
single filter invocation for the entire life of a single Git
command. This is achieved by using a packet format (pkt-line,
see technical/protocol-common.txt) based protocol over standard
input and standard output as follows. All packets, except for the
"*CONTENT" packets and the "0000" flush packet, are considered
text and therefore are terminated by a LF.
Git starts the filter when it encounters the first file
that needs to be cleaned or smudged. After the filter started
Git sends a welcome message ("git-filter-client"), a list of supported
protocol version numbers, and a flush packet. Git expects to read a welcome
response message ("git-filter-server"), exactly one protocol version number
from the previously sent list, and a flush packet. All further
communication will be based on the selected version. The remaining
protocol description below documents "version=2". Please note that
"version=42" in the example below does not exist and is only there
to illustrate how the protocol would look like with more than one
version.
After the version negotiation Git sends a list of all capabilities that
it supports and a flush packet. Git expects to read a list of desired
capabilities, which must be a subset of the supported capabilities list,
and a flush packet as response:
------------------------
packet: git> git-filter-client
packet: git> version=2
packet: git> version=42
packet: git> 0000
packet: git< git-filter-server
packet: git< version=2
packet: git< 0000
packet: git> capability=clean
packet: git> capability=smudge
packet: git> capability=not-yet-invented
packet: git> 0000
packet: git< capability=clean
packet: git< capability=smudge
packet: git< 0000
------------------------
Supported filter capabilities in version 2 are "clean" and
"smudge".
Afterwards Git sends a list of "key=value" pairs terminated with
a flush packet. The list will contain at least the filter command
(based on the supported capabilities) and the pathname of the file
to filter relative to the repository root. Right after the flush packet
Git sends the content split in zero or more pkt-line packets and a
flush packet to terminate content. Please note, that the filter
must not send any response before it received the content and the
final flush packet.
------------------------
packet: git> command=smudge
packet: git> pathname=path/testfile.dat
packet: git> 0000
packet: git> CONTENT
packet: git> 0000
------------------------
The filter is expected to respond with a list of "key=value" pairs
terminated with a flush packet. If the filter does not experience
problems then the list must contain a "success" status. Right after
these packets the filter is expected to send the content in zero
or more pkt-line packets and a flush packet at the end. Finally, a
second list of "key=value" pairs terminated with a flush packet
is expected. The filter can change the status in the second list
or keep the status as is with an empty list. Please note that the
empty list must be terminated with a flush packet regardless.
------------------------
packet: git< status=success
packet: git< 0000
packet: git< SMUDGED_CONTENT
packet: git< 0000
packet: git< 0000 # empty list, keep "status=success" unchanged!
------------------------
If the result content is empty then the filter is expected to respond
with a "success" status and a flush packet to signal the empty content.
------------------------
packet: git< status=success
packet: git< 0000
packet: git< 0000 # empty content!
packet: git< 0000 # empty list, keep "status=success" unchanged!
------------------------
In case the filter cannot or does not want to process the content,
it is expected to respond with an "error" status.
------------------------
packet: git< status=error
packet: git< 0000
------------------------
If the filter experiences an error during processing, then it can
send the status "error" after the content was (partially or
completely) sent.
------------------------
packet: git< status=success
packet: git< 0000
packet: git< HALF_WRITTEN_ERRONEOUS_CONTENT
packet: git< 0000
packet: git< status=error
packet: git< 0000
------------------------
In case the filter cannot or does not want to process the content
as well as any future content for the lifetime of the Git process,
then it is expected to respond with an "abort" status at any point
in the protocol.
------------------------
packet: git< status=abort
packet: git< 0000
------------------------
Git neither stops nor restarts the filter process in case the
"error"/"abort" status is set. However, Git sets its exit code
according to the `filter.<driver>.required` flag, mimicking the
behavior of the `filter.<driver>.clean` / `filter.<driver>.smudge`
mechanism.
If the filter dies during the communication or does not adhere to
the protocol then Git will stop the filter process and restart it
with the next file that needs to be processed. Depending on the
`filter.<driver>.required` flag Git will interpret that as error.
After the filter has processed a blob it is expected to wait for
the next "key=value" list containing a command. Git will close
the command pipe on exit. The filter is expected to detect EOF
and exit gracefully on its own. Git will wait until the filter
process has stopped.
A long running filter demo implementation can be found in
`contrib/long-running-filter/example.pl` located in the Git
core repository. If you develop your own long running filter
process then the `GIT_TRACE_PACKET` environment variables can be
very helpful for debugging (see linkgit:git[1]).
Please note that you cannot use an existing `filter.<driver>.clean`
or `filter.<driver>.smudge` command with `filter.<driver>.process`
because the former two use a different inter process communication
protocol than the latter one.
Interaction between checkin/checkout attributes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Просмотреть файл

@ -47,10 +47,10 @@ static int run_remote_archiver(int argc, const char **argv,
if (name_hint) {
const char *format = archive_format_from_filename(name_hint);
if (format)
packet_write(fd[1], "argument --format=%s\n", format);
packet_write_fmt(fd[1], "argument --format=%s\n", format);
}
for (i = 1; i < argc; i++)
packet_write(fd[1], "argument %s\n", argv[i]);
packet_write_fmt(fd[1], "argument %s\n", argv[i]);
packet_flush(fd[1]);
buf = packet_read_line(fd[0], NULL);

Просмотреть файл

@ -227,7 +227,7 @@ static int receive_pack_config(const char *var, const char *value, void *cb)
static void show_ref(const char *path, const unsigned char *sha1)
{
if (sent_capabilities) {
packet_write(1, "%s %s\n", sha1_to_hex(sha1), path);
packet_write_fmt(1, "%s %s\n", sha1_to_hex(sha1), path);
} else {
struct strbuf cap = STRBUF_INIT;
@ -242,7 +242,7 @@ static void show_ref(const char *path, const unsigned char *sha1)
if (advertise_push_options)
strbuf_addstr(&cap, " push-options");
strbuf_addf(&cap, " agent=%s", git_user_agent_sanitized());
packet_write(1, "%s %s%c%s\n",
packet_write_fmt(1, "%s %s%c%s\n",
sha1_to_hex(sha1), path, 0, cap.buf);
strbuf_release(&cap);
sent_capabilities = 1;

Просмотреть файл

@ -128,9 +128,9 @@ static void send_git_request(int stdin_fd, const char *serv, const char *repo,
const char *vhost)
{
if (!vhost)
packet_write(stdin_fd, "%s %s%c", serv, repo, 0);
packet_write_fmt(stdin_fd, "%s %s%c", serv, repo, 0);
else
packet_write(stdin_fd, "%s %s%chost=%s%c", serv, repo, 0,
packet_write_fmt(stdin_fd, "%s %s%chost=%s%c", serv, repo, 0,
vhost, 0);
}

Просмотреть файл

@ -88,11 +88,11 @@ int cmd_upload_archive(int argc, const char **argv, const char *prefix)
writer.git_cmd = 1;
if (start_command(&writer)) {
int err = errno;
packet_write(1, "NACK unable to spawn subprocess\n");
packet_write_fmt(1, "NACK unable to spawn subprocess\n");
die("upload-archive: %s", strerror(err));
}
packet_write(1, "ACK\n");
packet_write_fmt(1, "ACK\n");
packet_flush(1);
while (1) {

Просмотреть файл

@ -750,7 +750,7 @@ struct child_process *git_connect(int fd[2], const char *url,
* Note: Do not add any other headers here! Doing so
* will cause older git-daemon servers to crash.
*/
packet_write(fd[1],
packet_write_fmt(fd[1],
"%s %s%chost=%s%c",
prog, path, 0,
target_host, 0);

Просмотреть файл

@ -0,0 +1,128 @@
#!/usr/bin/perl
#
# Example implementation for the Git filter protocol version 2
# See Documentation/gitattributes.txt, section "Filter Protocol"
#
# Please note, this pass-thru filter is a minimal skeleton. No proper
# error handling was implemented.
#
use strict;
use warnings;
my $MAX_PACKET_CONTENT_SIZE = 65516;
sub packet_bin_read {
my $buffer;
my $bytes_read = read STDIN, $buffer, 4;
if ( $bytes_read == 0 ) {
# EOF - Git stopped talking to us!
exit();
}
elsif ( $bytes_read != 4 ) {
die "invalid packet: '$buffer'";
}
my $pkt_size = hex($buffer);
if ( $pkt_size == 0 ) {
return ( 1, "" );
}
elsif ( $pkt_size > 4 ) {
my $content_size = $pkt_size - 4;
$bytes_read = read STDIN, $buffer, $content_size;
if ( $bytes_read != $content_size ) {
die "invalid packet ($content_size bytes expected; $bytes_read bytes read)";
}
return ( 0, $buffer );
}
else {
die "invalid packet size: $pkt_size";
}
}
sub packet_txt_read {
my ( $res, $buf ) = packet_bin_read();
unless ( $buf =~ s/\n$// ) {
die "A non-binary line MUST be terminated by an LF.";
}
return ( $res, $buf );
}
sub packet_bin_write {
my $buf = shift;
print STDOUT sprintf( "%04x", length($buf) + 4 );
print STDOUT $buf;
STDOUT->flush();
}
sub packet_txt_write {
packet_bin_write( $_[0] . "\n" );
}
sub packet_flush {
print STDOUT sprintf( "%04x", 0 );
STDOUT->flush();
}
( packet_txt_read() eq ( 0, "git-filter-client" ) ) || die "bad initialize";
( packet_txt_read() eq ( 0, "version=2" ) ) || die "bad version";
( packet_bin_read() eq ( 1, "" ) ) || die "bad version end";
packet_txt_write("git-filter-server");
packet_txt_write("version=2");
packet_flush();
( packet_txt_read() eq ( 0, "capability=clean" ) ) || die "bad capability";
( packet_txt_read() eq ( 0, "capability=smudge" ) ) || die "bad capability";
( packet_bin_read() eq ( 1, "" ) ) || die "bad capability end";
packet_txt_write("capability=clean");
packet_txt_write("capability=smudge");
packet_flush();
while (1) {
my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;
my ($pathname) = packet_txt_read() =~ /^pathname=([^=]+)$/;
packet_bin_read();
my $input = "";
{
binmode(STDIN);
my $buffer;
my $done = 0;
while ( !$done ) {
( $done, $buffer ) = packet_bin_read();
$input .= $buffer;
}
}
my $output;
if ( $command eq "clean" ) {
### Perform clean here ###
$output = $input;
}
elsif ( $command eq "smudge" ) {
### Perform smudge here ###
$output = $input;
}
else {
die "bad command '$command'";
}
packet_txt_write("status=success");
packet_flush();
while ( length($output) > 0 ) {
my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );
packet_bin_write($packet);
if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {
$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );
}
else {
$output = "";
}
}
packet_flush(); # flush content!
packet_flush(); # empty list, keep "status=success" unchanged!
}

376
convert.c
Просмотреть файл

@ -3,6 +3,7 @@
#include "run-command.h"
#include "quote.h"
#include "sigchain.h"
#include "pkt-line.h"
/*
* convert.c - convert a file when checking it out and checking it in.
@ -416,7 +417,7 @@ static int filter_buffer_or_fd(int in, int out, void *data)
child_process.out = out;
if (start_command(&child_process))
return error("cannot fork to run external filter %s", params->cmd);
return error("cannot fork to run external filter '%s'", params->cmd);
sigchain_push(SIGPIPE, SIG_IGN);
@ -434,19 +435,19 @@ static int filter_buffer_or_fd(int in, int out, void *data)
if (close(child_process.in))
write_err = 1;
if (write_err)
error("cannot feed the input to external filter %s", params->cmd);
error("cannot feed the input to external filter '%s'", params->cmd);
sigchain_pop(SIGPIPE);
status = finish_command(&child_process);
if (status)
error("external filter %s failed %d", params->cmd, status);
error("external filter '%s' failed %d", params->cmd, status);
strbuf_release(&cmd);
return (write_err || status);
}
static int apply_filter(const char *path, const char *src, size_t len, int fd,
static int apply_single_file_filter(const char *path, const char *src, size_t len, int fd,
struct strbuf *dst, const char *cmd)
{
/*
@ -455,17 +456,11 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,
*
* (child --> cmd) --> us
*/
int ret = 1;
int err = 0;
struct strbuf nbuf = STRBUF_INIT;
struct async async;
struct filter_params params;
if (!cmd || !*cmd)
return 0;
if (!dst)
return 1;
memset(&async, 0, sizeof(async));
async.proc = filter_buffer_or_fd;
async.data = &params;
@ -481,23 +476,304 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,
return 0; /* error was already reported */
if (strbuf_read(&nbuf, async.out, len) < 0) {
error("read from external filter %s failed", cmd);
ret = 0;
err = error("read from external filter '%s' failed", cmd);
}
if (close(async.out)) {
error("read from external filter %s failed", cmd);
ret = 0;
err = error("read from external filter '%s' failed", cmd);
}
if (finish_async(&async)) {
error("external filter %s failed", cmd);
ret = 0;
err = error("external filter '%s' failed", cmd);
}
if (ret) {
if (!err) {
strbuf_swap(dst, &nbuf);
}
strbuf_release(&nbuf);
return ret;
return !err;
}
#define CAP_CLEAN (1u<<0)
#define CAP_SMUDGE (1u<<1)
struct cmd2process {
struct hashmap_entry ent; /* must be the first member! */
unsigned int supported_capabilities;
const char *cmd;
struct child_process process;
};
static int cmd_process_map_initialized;
static struct hashmap cmd_process_map;
static int cmd2process_cmp(const struct cmd2process *e1,
const struct cmd2process *e2,
const void *unused)
{
return strcmp(e1->cmd, e2->cmd);
}
static struct cmd2process *find_multi_file_filter_entry(struct hashmap *hashmap, const char *cmd)
{
struct cmd2process key;
hashmap_entry_init(&key, strhash(cmd));
key.cmd = cmd;
return hashmap_get(hashmap, &key, NULL);
}
static int packet_write_list(int fd, const char *line, ...)
{
va_list args;
int err;
va_start(args, line);
for (;;) {
if (!line)
break;
if (strlen(line) > LARGE_PACKET_DATA_MAX)
return -1;
err = packet_write_fmt_gently(fd, "%s\n", line);
if (err)
return err;
line = va_arg(args, const char*);
}
va_end(args);
return packet_flush_gently(fd);
}
static void read_multi_file_filter_status(int fd, struct strbuf *status)
{
struct strbuf **pair;
char *line;
for (;;) {
line = packet_read_line(fd, NULL);
if (!line)
break;
pair = strbuf_split_str(line, '=', 2);
if (pair[0] && pair[0]->len && pair[1]) {
/* the last "status=<foo>" line wins */
if (!strcmp(pair[0]->buf, "status=")) {
strbuf_reset(status);
strbuf_addbuf(status, pair[1]);
}
}
strbuf_list_free(pair);
}
}
static void kill_multi_file_filter(struct hashmap *hashmap, struct cmd2process *entry)
{
if (!entry)
return;
entry->process.clean_on_exit = 0;
kill(entry->process.pid, SIGTERM);
finish_command(&entry->process);
hashmap_remove(hashmap, entry, NULL);
free(entry);
}
static void stop_multi_file_filter(struct child_process *process)
{
sigchain_push(SIGPIPE, SIG_IGN);
/* Closing the pipe signals the filter to initiate a shutdown. */
close(process->in);
close(process->out);
sigchain_pop(SIGPIPE);
/* Finish command will wait until the shutdown is complete. */
finish_command(process);
}
static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, const char *cmd)
{
int err;
struct cmd2process *entry;
struct child_process *process;
const char *argv[] = { cmd, NULL };
struct string_list cap_list = STRING_LIST_INIT_NODUP;
char *cap_buf;
const char *cap_name;
entry = xmalloc(sizeof(*entry));
entry->cmd = cmd;
entry->supported_capabilities = 0;
process = &entry->process;
child_process_init(process);
process->argv = argv;
process->use_shell = 1;
process->in = -1;
process->out = -1;
process->clean_on_exit = 1;
process->clean_on_exit_handler = stop_multi_file_filter;
if (start_command(process)) {
error("cannot fork to run external filter '%s'", cmd);
return NULL;
}
hashmap_entry_init(entry, strhash(cmd));
sigchain_push(SIGPIPE, SIG_IGN);
err = packet_write_list(process->in, "git-filter-client", "version=2", NULL);
if (err)
goto done;
err = strcmp(packet_read_line(process->out, NULL), "git-filter-server");
if (err) {
error("external filter '%s' does not support filter protocol version 2", cmd);
goto done;
}
err = strcmp(packet_read_line(process->out, NULL), "version=2");
if (err)
goto done;
err = packet_read_line(process->out, NULL) != NULL;
if (err)
goto done;
err = packet_write_list(process->in, "capability=clean", "capability=smudge", NULL);
for (;;) {
cap_buf = packet_read_line(process->out, NULL);
if (!cap_buf)
break;
string_list_split_in_place(&cap_list, cap_buf, '=', 1);
if (cap_list.nr != 2 || strcmp(cap_list.items[0].string, "capability"))
continue;
cap_name = cap_list.items[1].string;
if (!strcmp(cap_name, "clean")) {
entry->supported_capabilities |= CAP_CLEAN;
} else if (!strcmp(cap_name, "smudge")) {
entry->supported_capabilities |= CAP_SMUDGE;
} else {
warning(
"external filter '%s' requested unsupported filter capability '%s'",
cmd, cap_name
);
}
string_list_clear(&cap_list, 0);
}
done:
sigchain_pop(SIGPIPE);
if (err || errno == EPIPE) {
error("initialization for external filter '%s' failed", cmd);
kill_multi_file_filter(hashmap, entry);
return NULL;
}
hashmap_add(hashmap, entry);
return entry;
}
static int apply_multi_file_filter(const char *path, const char *src, size_t len,
int fd, struct strbuf *dst, const char *cmd,
const unsigned int wanted_capability)
{
int err;
struct cmd2process *entry;
struct child_process *process;
struct strbuf nbuf = STRBUF_INIT;
struct strbuf filter_status = STRBUF_INIT;
const char *filter_type;
if (!cmd_process_map_initialized) {
cmd_process_map_initialized = 1;
hashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);
entry = NULL;
} else {
entry = find_multi_file_filter_entry(&cmd_process_map, cmd);
}
fflush(NULL);
if (!entry) {
entry = start_multi_file_filter(&cmd_process_map, cmd);
if (!entry)
return 0;
}
process = &entry->process;
if (!(wanted_capability & entry->supported_capabilities))
return 0;
if (CAP_CLEAN & wanted_capability)
filter_type = "clean";
else if (CAP_SMUDGE & wanted_capability)
filter_type = "smudge";
else
die("unexpected filter type");
sigchain_push(SIGPIPE, SIG_IGN);
assert(strlen(filter_type) < LARGE_PACKET_DATA_MAX - strlen("command=\n"));
err = packet_write_fmt_gently(process->in, "command=%s\n", filter_type);
if (err)
goto done;
err = strlen(path) > LARGE_PACKET_DATA_MAX - strlen("pathname=\n");
if (err) {
error("path name too long for external filter");
goto done;
}
err = packet_write_fmt_gently(process->in, "pathname=%s\n", path);
if (err)
goto done;
err = packet_flush_gently(process->in);
if (err)
goto done;
if (fd >= 0)
err = write_packetized_from_fd(fd, process->in);
else
err = write_packetized_from_buf(src, len, process->in);
if (err)
goto done;
read_multi_file_filter_status(process->out, &filter_status);
err = strcmp(filter_status.buf, "success");
if (err)
goto done;
err = read_packetized_to_strbuf(process->out, &nbuf) < 0;
if (err)
goto done;
read_multi_file_filter_status(process->out, &filter_status);
err = strcmp(filter_status.buf, "success");
done:
sigchain_pop(SIGPIPE);
if (err || errno == EPIPE) {
if (!strcmp(filter_status.buf, "error")) {
/* The filter signaled a problem with the file. */
} else if (!strcmp(filter_status.buf, "abort")) {
/*
* The filter signaled a permanent problem. Don't try to filter
* files with the same command for the lifetime of the current
* Git process.
*/
entry->supported_capabilities &= ~wanted_capability;
} else {
/*
* Something went wrong with the protocol filter.
* Force shutdown and restart if another blob requires filtering.
*/
error("external filter '%s' failed", cmd);
kill_multi_file_filter(&cmd_process_map, entry);
}
} else {
strbuf_swap(dst, &nbuf);
}
strbuf_release(&nbuf);
return !err;
}
static struct convert_driver {
@ -505,9 +781,35 @@ static struct convert_driver {
struct convert_driver *next;
const char *smudge;
const char *clean;
const char *process;
int required;
} *user_convert, **user_convert_tail;
static int apply_filter(const char *path, const char *src, size_t len,
int fd, struct strbuf *dst, struct convert_driver *drv,
const unsigned int wanted_capability)
{
const char *cmd = NULL;
if (!drv)
return 0;
if (!dst)
return 1;
if ((CAP_CLEAN & wanted_capability) && !drv->process && drv->clean)
cmd = drv->clean;
else if ((CAP_SMUDGE & wanted_capability) && !drv->process && drv->smudge)
cmd = drv->smudge;
if (cmd && *cmd)
return apply_single_file_filter(path, src, len, fd, dst, cmd);
else if (drv->process && *drv->process)
return apply_multi_file_filter(path, src, len, fd, dst, drv->process, wanted_capability);
return 0;
}
static int read_convert_config(const char *var, const char *value, void *cb)
{
const char *key, *name;
@ -545,6 +847,9 @@ static int read_convert_config(const char *var, const char *value, void *cb)
if (!strcmp("clean", key))
return git_config_string(&drv->clean, var, value);
if (!strcmp("process", key))
return git_config_string(&drv->process, var, value);
if (!strcmp("required", key)) {
drv->required = git_config_bool(var, value);
return 0;
@ -846,7 +1151,7 @@ int would_convert_to_git_filter_fd(const char *path)
if (!ca.drv->required)
return 0;
return apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);
return apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN);
}
const char *get_convert_attr_ascii(const char *path)
@ -879,18 +1184,12 @@ int convert_to_git(const char *path, const char *src, size_t len,
struct strbuf *dst, enum safe_crlf checksafe)
{
int ret = 0;
const char *filter = NULL;
int required = 0;
struct conv_attrs ca;
convert_attrs(&ca, path);
if (ca.drv) {
filter = ca.drv->clean;
required = ca.drv->required;
}
ret |= apply_filter(path, src, len, -1, dst, filter);
if (!ret && required)
ret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN);
if (!ret && ca.drv && ca.drv->required)
die("%s: clean filter '%s' failed", path, ca.drv->name);
if (ret && dst) {
@ -912,9 +1211,9 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,
convert_attrs(&ca, path);
assert(ca.drv);
assert(ca.drv->clean);
assert(ca.drv->clean || ca.drv->process);
if (!apply_filter(path, NULL, 0, fd, dst, ca.drv->clean))
if (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN))
die("%s: clean filter '%s' failed", path, ca.drv->name);
crlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);
@ -926,15 +1225,9 @@ static int convert_to_working_tree_internal(const char *path, const char *src,
int normalizing)
{
int ret = 0, ret_filter = 0;
const char *filter = NULL;
int required = 0;
struct conv_attrs ca;
convert_attrs(&ca, path);
if (ca.drv) {
filter = ca.drv->smudge;
required = ca.drv->required;
}
ret |= ident_to_worktree(path, src, len, dst, ca.ident);
if (ret) {
@ -943,9 +1236,10 @@ static int convert_to_working_tree_internal(const char *path, const char *src,
}
/*
* CRLF conversion can be skipped if normalizing, unless there
* is a smudge filter. The filter might expect CRLFs.
* is a smudge or process filter (even if the process filter doesn't
* support smudge). The filters might expect CRLFs.
*/
if (filter || !normalizing) {
if ((ca.drv && (ca.drv->smudge || ca.drv->process)) || !normalizing) {
ret |= crlf_to_worktree(path, src, len, dst, ca.crlf_action);
if (ret) {
src = dst->buf;
@ -953,8 +1247,8 @@ static int convert_to_working_tree_internal(const char *path, const char *src,
}
}
ret_filter = apply_filter(path, src, len, -1, dst, filter);
if (!ret_filter && required)
ret_filter = apply_filter(path, src, len, -1, dst, ca.drv, CAP_SMUDGE);
if (!ret_filter && ca.drv && ca.drv->required)
die("%s: smudge filter %s failed", path, ca.drv->name);
return ret | ret_filter;
@ -1406,7 +1700,7 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s
struct stream_filter *filter = NULL;
convert_attrs(&ca, path);
if (ca.drv && (ca.drv->smudge || ca.drv->clean))
if (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))
return NULL;
if (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)

Просмотреть файл

@ -298,7 +298,7 @@ static int daemon_error(const char *dir, const char *msg)
{
if (!informative_errors)
msg = "access denied or repository not exported";
packet_write(1, "ERR %s: %s", msg, dir);
packet_write_fmt(1, "ERR %s: %s", msg, dir);
return -1;
}

Просмотреть файл

@ -464,7 +464,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg)
hdr_str(hdr, content_type, buf.buf);
end_headers(hdr);
packet_write(1, "# service=git-%s\n", svc->name);
packet_write_fmt(1, "# service=git-%s\n", svc->name);
packet_flush(1);
argv[0] = svc->name;

Просмотреть файл

@ -91,16 +91,34 @@ void packet_flush(int fd)
write_or_die(fd, "0000", 4);
}
int packet_flush_gently(int fd)
{
packet_trace("0000", 4, 1);
if (write_in_full(fd, "0000", 4) == 4)
return 0;
return error("flush packet write failed");
}
void packet_buf_flush(struct strbuf *buf)
{
packet_trace("0000", 4, 1);
strbuf_add(buf, "0000", 4);
}
#define hex(a) (hexchar[(a) & 15])
static void format_packet(struct strbuf *out, const char *fmt, va_list args)
static void set_packet_header(char *buf, const int size)
{
static char hexchar[] = "0123456789abcdef";
#define hex(a) (hexchar[(a) & 15])
buf[0] = hex(size >> 12);
buf[1] = hex(size >> 8);
buf[2] = hex(size >> 4);
buf[3] = hex(size);
#undef hex
}
static void format_packet(struct strbuf *out, const char *fmt, va_list args)
{
size_t orig_len, n;
orig_len = out->len;
@ -111,23 +129,63 @@ static void format_packet(struct strbuf *out, const char *fmt, va_list args)
if (n > LARGE_PACKET_MAX)
die("protocol error: impossibly long line");
out->buf[orig_len + 0] = hex(n >> 12);
out->buf[orig_len + 1] = hex(n >> 8);
out->buf[orig_len + 2] = hex(n >> 4);
out->buf[orig_len + 3] = hex(n);
set_packet_header(&out->buf[orig_len], n);
packet_trace(out->buf + orig_len + 4, n - 4, 1);
}
void packet_write(int fd, const char *fmt, ...)
static int packet_write_fmt_1(int fd, int gently,
const char *fmt, va_list args)
{
struct strbuf buf = STRBUF_INIT;
ssize_t count;
format_packet(&buf, fmt, args);
count = write_in_full(fd, buf.buf, buf.len);
if (count == buf.len)
return 0;
if (!gently) {
check_pipe(errno);
die_errno("packet write with format failed");
}
return error("packet write with format failed");
}
void packet_write_fmt(int fd, const char *fmt, ...)
{
static struct strbuf buf = STRBUF_INIT;
va_list args;
strbuf_reset(&buf);
va_start(args, fmt);
format_packet(&buf, fmt, args);
packet_write_fmt_1(fd, 0, fmt, args);
va_end(args);
write_or_die(fd, buf.buf, buf.len);
}
int packet_write_fmt_gently(int fd, const char *fmt, ...)
{
int status;
va_list args;
va_start(args, fmt);
status = packet_write_fmt_1(fd, 1, fmt, args);
va_end(args);
return status;
}
static int packet_write_gently(const int fd_out, const char *buf, size_t size)
{
static char packet_write_buffer[LARGE_PACKET_MAX];
size_t packet_size;
if (size > sizeof(packet_write_buffer) - 4)
return error("packet write failed - data exceeds max packet size");
packet_trace(buf, size, 1);
packet_size = size + 4;
set_packet_header(packet_write_buffer, packet_size);
memcpy(packet_write_buffer + 4, buf, size);
if (write_in_full(fd_out, packet_write_buffer, packet_size) == packet_size)
return 0;
return error("packet write failed");
}
void packet_buf_write(struct strbuf *buf, const char *fmt, ...)
@ -139,6 +197,46 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)
va_end(args);
}
int write_packetized_from_fd(int fd_in, int fd_out)
{
static char buf[LARGE_PACKET_DATA_MAX];
int err = 0;
ssize_t bytes_to_write;
while (!err) {
bytes_to_write = xread(fd_in, buf, sizeof(buf));
if (bytes_to_write < 0)
return COPY_READ_ERROR;
if (bytes_to_write == 0)
break;
err = packet_write_gently(fd_out, buf, bytes_to_write);
}
if (!err)
err = packet_flush_gently(fd_out);
return err;
}
int write_packetized_from_buf(const char *src_in, size_t len, int fd_out)
{
int err = 0;
size_t bytes_written = 0;
size_t bytes_to_write;
while (!err) {
if ((len - bytes_written) > LARGE_PACKET_DATA_MAX)
bytes_to_write = LARGE_PACKET_DATA_MAX;
else
bytes_to_write = len - bytes_written;
if (bytes_to_write == 0)
break;
err = packet_write_gently(fd_out, src_in + bytes_written, bytes_to_write);
bytes_written += bytes_to_write;
}
if (!err)
err = packet_flush_gently(fd_out);
return err;
}
static int get_packet_data(int fd, char **src_buf, size_t *src_size,
void *dst, unsigned size, int options)
{
@ -229,3 +327,35 @@ char *packet_read_line_buf(char **src, size_t *src_len, int *dst_len)
{
return packet_read_line_generic(-1, src, src_len, dst_len);
}
ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out)
{
int packet_len;
size_t orig_len = sb_out->len;
size_t orig_alloc = sb_out->alloc;
for (;;) {
strbuf_grow(sb_out, LARGE_PACKET_DATA_MAX);
packet_len = packet_read(fd_in, NULL, NULL,
/* strbuf_grow() above always allocates one extra byte to
* store a '\0' at the end of the string. packet_read()
* writes a '\0' extra byte at the end, too. Let it know
* that there is already room for the extra byte.
*/
sb_out->buf + sb_out->len, LARGE_PACKET_DATA_MAX+1,
PACKET_READ_GENTLE_ON_EOF);
if (packet_len <= 0)
break;
sb_out->len += packet_len;
}
if (packet_len < 0) {
if (orig_alloc == 0)
strbuf_release(sb_out);
else
strbuf_setlen(sb_out, orig_len);
return packet_len;
}
return sb_out->len - orig_len;
}

Просмотреть файл

@ -20,9 +20,13 @@
* side can't, we stay with pure read/write interfaces.
*/
void packet_flush(int fd);
void packet_write(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
void packet_buf_flush(struct strbuf *buf);
void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
int packet_flush_gently(int fd);
int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
int write_packetized_from_fd(int fd_in, int fd_out);
int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);
/*
* Read a packetized line into the buffer, which must be at least size bytes
@ -75,8 +79,14 @@ char *packet_read_line(int fd, int *size);
*/
char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);
/*
* Reads a stream of variable sized packets until a flush packet is detected.
*/
ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);
#define DEFAULT_PACKET_MAX 1000
#define LARGE_PACKET_MAX 65520
#define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)
extern char packet_buffer[LARGE_PACKET_MAX];
#endif

Просмотреть файл

@ -21,6 +21,7 @@ void child_process_clear(struct child_process *child)
struct child_to_clean {
pid_t pid;
struct child_process *process;
struct child_to_clean *next;
};
static struct child_to_clean *children_to_clean;
@ -31,6 +32,18 @@ static void cleanup_children(int sig, int in_signal)
while (children_to_clean) {
struct child_to_clean *p = children_to_clean;
children_to_clean = p->next;
if (p->process && !in_signal) {
struct child_process *process = p->process;
if (process->clean_on_exit_handler) {
trace_printf(
"trace: run_command: running exit handler for pid %"
PRIuMAX, (uintmax_t)p->pid
);
process->clean_on_exit_handler(process);
}
}
kill(p->pid, sig);
if (!in_signal)
free(p);
@ -49,10 +62,11 @@ static void cleanup_children_on_exit(void)
cleanup_children(SIGTERM, 0);
}
static void mark_child_for_cleanup(pid_t pid)
static void mark_child_for_cleanup(pid_t pid, struct child_process *process)
{
struct child_to_clean *p = xmalloc(sizeof(*p));
p->pid = pid;
p->process = process;
p->next = children_to_clean;
children_to_clean = p;
@ -422,7 +436,7 @@ fail_pipe:
if (cmd->pid < 0)
error_errno("cannot fork() for %s", cmd->argv[0]);
else if (cmd->clean_on_exit)
mark_child_for_cleanup(cmd->pid);
mark_child_for_cleanup(cmd->pid, cmd);
/*
* Wait for child's execvp. If the execvp succeeds (or if fork()
@ -483,7 +497,7 @@ fail_pipe:
if (cmd->pid < 0 && (!cmd->silent_exec_failure || errno != ENOENT))
error_errno("cannot spawn %s", cmd->argv[0]);
if (cmd->clean_on_exit && cmd->pid >= 0)
mark_child_for_cleanup(cmd->pid);
mark_child_for_cleanup(cmd->pid, cmd);
argv_array_clear(&nargv);
cmd->argv = sargv;
@ -634,7 +648,7 @@ int in_async(void)
return !pthread_equal(main_thread, pthread_self());
}
void NORETURN async_exit(int code)
static void NORETURN async_exit(int code)
{
pthread_exit((void *)(intptr_t)code);
}
@ -684,13 +698,26 @@ int in_async(void)
return process_is_async;
}
void NORETURN async_exit(int code)
static void NORETURN async_exit(int code)
{
exit(code);
}
#endif
void check_pipe(int err)
{
if (err == EPIPE) {
if (in_async())
async_exit(141);
signal(SIGPIPE, SIG_DFL);
raise(SIGPIPE);
/* Should never happen, but just in case... */
exit(141);
}
}
int start_async(struct async *async)
{
int need_in, need_out;
@ -752,7 +779,7 @@ int start_async(struct async *async)
exit(!!async->proc(proc_in, proc_out, async->data));
}
mark_child_for_cleanup(async->pid);
mark_child_for_cleanup(async->pid, NULL);
if (need_in)
close(fdin[0]);

Просмотреть файл

@ -43,6 +43,8 @@ struct child_process {
unsigned stdout_to_stderr:1;
unsigned use_shell:1;
unsigned clean_on_exit:1;
void (*clean_on_exit_handler)(struct child_process *process);
void *clean_on_exit_handler_cbdata;
};
#define CHILD_PROCESS_INIT { NULL, ARGV_ARRAY_INIT, ARGV_ARRAY_INIT }
@ -139,7 +141,7 @@ struct async {
int start_async(struct async *async);
int finish_async(struct async *async);
int in_async(void);
void NORETURN async_exit(int code);
void check_pipe(int err);
/**
* This callback should initialize the child process and preload the

Просмотреть файл

@ -338,7 +338,7 @@ static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *c
{
int fd = *(int *)cb;
if (graft->nr_parent == -1)
packet_write(fd, "shallow %s\n", oid_to_hex(&graft->oid));
packet_write_fmt(fd, "shallow %s\n", oid_to_hex(&graft->oid));
return 0;
}

Просмотреть файл

@ -4,13 +4,72 @@ test_description='blob conversion via gitattributes'
. ./test-lib.sh
cat <<EOF >rot13.sh
TEST_ROOT="$(pwd)"
cat <<EOF >"$TEST_ROOT/rot13.sh"
#!$SHELL_PATH
tr \
'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' \
'nopqrstuvwxyzabcdefghijklmNOPQRSTUVWXYZABCDEFGHIJKLM'
EOF
chmod +x rot13.sh
chmod +x "$TEST_ROOT/rot13.sh"
generate_random_characters () {
LEN=$1
NAME=$2
test-genrandom some-seed $LEN |
perl -pe "s/./chr((ord($&) % 26) + ord('a'))/sge" >"$TEST_ROOT/$NAME"
}
file_size () {
cat "$1" | wc -c | sed "s/^[ ]*//"
}
filter_git () {
rm -f rot13-filter.log &&
git "$@" 2>git-stderr.log &&
rm -f git-stderr.log
}
# Compare two files and ensure that `clean` and `smudge` respectively are
# called at least once if specified in the `expect` file. The actual
# invocation count is not relevant because their number can vary.
# c.f. http://public-inbox.org/git/xmqqshv18i8i.fsf@gitster.mtv.corp.google.com/
test_cmp_count () {
expect=$1
actual=$2
for FILE in "$expect" "$actual"
do
sort "$FILE" | uniq -c | sed "s/^[ ]*//" |
sed "s/^\([0-9]\) IN: clean/x IN: clean/" |
sed "s/^\([0-9]\) IN: smudge/x IN: smudge/" >"$FILE.tmp" &&
mv "$FILE.tmp" "$FILE"
done &&
test_cmp "$expect" "$actual"
}
# Compare two files but exclude all `clean` invocations because Git can
# call `clean` zero or more times.
# c.f. http://public-inbox.org/git/xmqqshv18i8i.fsf@gitster.mtv.corp.google.com/
test_cmp_exclude_clean () {
expect=$1
actual=$2
for FILE in "$expect" "$actual"
do
grep -v "IN: clean" "$FILE" >"$FILE.tmp" &&
mv "$FILE.tmp" "$FILE"
done &&
test_cmp "$expect" "$actual"
}
# Check that the contents of two files are equal and that their rot13 version
# is equal to the committed content.
test_cmp_committed_rot13 () {
test_cmp "$1" "$2" &&
"$TEST_ROOT/rot13.sh" <"$1" >expected &&
git cat-file blob :"$2" >actual &&
test_cmp expected actual
}
test_expect_success setup '
git config filter.rot13.smudge ./rot13.sh &&
@ -31,15 +90,18 @@ test_expect_success setup '
cat test >test.i &&
git add test test.t test.i &&
rm -f test test.t test.i &&
git checkout -- test test.t test.i
git checkout -- test test.t test.i &&
echo "content-test2" >test2.o &&
echo "content-test3 - filename with special characters" >"test3 '\''sq'\'',\$x.o"
'
script='s/^\$Id: \([0-9a-f]*\) \$/\1/p'
test_expect_success check '
cmp test.o test &&
cmp test.o test.t &&
test_cmp test.o test &&
test_cmp test.o test.t &&
# ident should be stripped in the repository
git diff --raw --exit-code :test :test.i &&
@ -47,10 +109,10 @@ test_expect_success check '
embedded=$(sed -ne "$script" test.i) &&
test "z$id" = "z$embedded" &&
git cat-file blob :test.t > test.r &&
git cat-file blob :test.t >test.r &&
./rot13.sh < test.o > test.t &&
cmp test.r test.t
./rot13.sh <test.o >test.t &&
test_cmp test.r test.t
'
# If an expanded ident ever gets into the repository, we want to make sure that
@ -130,7 +192,7 @@ test_expect_success 'filter shell-escaped filenames' '
# delete the files and check them out again, using a smudge filter
# that will count the args and echo the command-line back to us
git config filter.argc.smudge "sh ./argc.sh %f" &&
test_config filter.argc.smudge "sh ./argc.sh %f" &&
rm "$normal" "$special" &&
git checkout -- "$normal" "$special" &&
@ -141,7 +203,7 @@ test_expect_success 'filter shell-escaped filenames' '
test_cmp expect "$special" &&
# do the same thing, but with more args in the filter expression
git config filter.argc.smudge "sh ./argc.sh %f --my-extra-arg" &&
test_config filter.argc.smudge "sh ./argc.sh %f --my-extra-arg" &&
rm "$normal" "$special" &&
git checkout -- "$normal" "$special" &&
@ -154,9 +216,9 @@ test_expect_success 'filter shell-escaped filenames' '
'
test_expect_success 'required filter should filter data' '
git config filter.required.smudge ./rot13.sh &&
git config filter.required.clean ./rot13.sh &&
git config filter.required.required true &&
test_config filter.required.smudge ./rot13.sh &&
test_config filter.required.clean ./rot13.sh &&
test_config filter.required.required true &&
echo "*.r filter=required" >.gitattributes &&
@ -165,17 +227,17 @@ test_expect_success 'required filter should filter data' '
rm -f test.r &&
git checkout -- test.r &&
cmp test.o test.r &&
test_cmp test.o test.r &&
./rot13.sh <test.o >expected &&
git cat-file blob :test.r >actual &&
cmp expected actual
test_cmp expected actual
'
test_expect_success 'required filter smudge failure' '
git config filter.failsmudge.smudge false &&
git config filter.failsmudge.clean cat &&
git config filter.failsmudge.required true &&
test_config filter.failsmudge.smudge false &&
test_config filter.failsmudge.clean cat &&
test_config filter.failsmudge.required true &&
echo "*.fs filter=failsmudge" >.gitattributes &&
@ -186,9 +248,9 @@ test_expect_success 'required filter smudge failure' '
'
test_expect_success 'required filter clean failure' '
git config filter.failclean.smudge cat &&
git config filter.failclean.clean false &&
git config filter.failclean.required true &&
test_config filter.failclean.smudge cat &&
test_config filter.failclean.clean false &&
test_config filter.failclean.required true &&
echo "*.fc filter=failclean" >.gitattributes &&
@ -197,8 +259,8 @@ test_expect_success 'required filter clean failure' '
'
test_expect_success 'filtering large input to small output should use little memory' '
git config filter.devnull.clean "cat >/dev/null" &&
git config filter.devnull.required true &&
test_config filter.devnull.clean "cat >/dev/null" &&
test_config filter.devnull.required true &&
for i in $(test_seq 1 30); do printf "%1048576d" 1; done >30MB &&
echo "30MB filter=devnull" >.gitattributes &&
GIT_MMAP_LIMIT=1m GIT_ALLOC_LIMIT=1m git add 30MB
@ -207,7 +269,7 @@ test_expect_success 'filtering large input to small output should use little mem
test_expect_success 'filter that does not read is fine' '
test-genrandom foo $((128 * 1024 + 1)) >big &&
echo "big filter=epipe" >.gitattributes &&
git config filter.epipe.clean "echo xyzzy" &&
test_config filter.epipe.clean "echo xyzzy" &&
git add big &&
git cat-file blob :big >actual &&
echo xyzzy >expect &&
@ -215,20 +277,20 @@ test_expect_success 'filter that does not read is fine' '
'
test_expect_success EXPENSIVE 'filter large file' '
git config filter.largefile.smudge cat &&
git config filter.largefile.clean cat &&
test_config filter.largefile.smudge cat &&
test_config filter.largefile.clean cat &&
for i in $(test_seq 1 2048); do printf "%1048576d" 1; done >2GB &&
echo "2GB filter=largefile" >.gitattributes &&
git add 2GB 2>err &&
! test -s err &&
test_must_be_empty err &&
rm -f 2GB &&
git checkout -- 2GB 2>err &&
! test -s err
test_must_be_empty err
'
test_expect_success "filter: clean empty file" '
git config filter.in-repo-header.clean "echo cleaned && cat" &&
git config filter.in-repo-header.smudge "sed 1d" &&
test_config filter.in-repo-header.clean "echo cleaned && cat" &&
test_config filter.in-repo-header.smudge "sed 1d" &&
echo "empty-in-worktree filter=in-repo-header" >>.gitattributes &&
>empty-in-worktree &&
@ -240,8 +302,8 @@ test_expect_success "filter: clean empty file" '
'
test_expect_success "filter: smudge empty file" '
git config filter.empty-in-repo.clean "cat >/dev/null" &&
git config filter.empty-in-repo.smudge "echo smudged && cat" &&
test_config filter.empty-in-repo.clean "cat >/dev/null" &&
test_config filter.empty-in-repo.smudge "echo smudged && cat" &&
echo "empty-in-repo filter=empty-in-repo" >>.gitattributes &&
echo dead data walking >empty-in-repo &&
@ -279,4 +341,380 @@ test_expect_success 'diff does not reuse worktree files that need cleaning' '
test_line_count = 0 count
'
test_expect_success PERL 'required process filter should filter data' '
test_config_global filter.protocol.process "$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge" &&
test_config_global filter.protocol.required true &&
rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "git-stderr.log" >.gitignore &&
echo "*.r filter=protocol" >.gitattributes &&
git add . &&
git commit . -m "test commit 1" &&
git branch empty-branch &&
cp "$TEST_ROOT/test.o" test.r &&
cp "$TEST_ROOT/test2.o" test2.r &&
mkdir testsubdir &&
cp "$TEST_ROOT/test3 '\''sq'\'',\$x.o" "testsubdir/test3 '\''sq'\'',\$x.r" &&
>test4-empty.r &&
S=$(file_size test.r) &&
S2=$(file_size test2.r) &&
S3=$(file_size "testsubdir/test3 '\''sq'\'',\$x.r") &&
filter_git add . &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: clean test.r $S [OK] -- OUT: $S . [OK]
IN: clean test2.r $S2 [OK] -- OUT: $S2 . [OK]
IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]
IN: clean testsubdir/test3 '\''sq'\'',\$x.r $S3 [OK] -- OUT: $S3 . [OK]
STOP
EOF
test_cmp_count expected.log rot13-filter.log &&
filter_git commit . -m "test commit 2" &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: clean test.r $S [OK] -- OUT: $S . [OK]
IN: clean test2.r $S2 [OK] -- OUT: $S2 . [OK]
IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]
IN: clean testsubdir/test3 '\''sq'\'',\$x.r $S3 [OK] -- OUT: $S3 . [OK]
IN: clean test.r $S [OK] -- OUT: $S . [OK]
IN: clean test2.r $S2 [OK] -- OUT: $S2 . [OK]
IN: clean test4-empty.r 0 [OK] -- OUT: 0 [OK]
IN: clean testsubdir/test3 '\''sq'\'',\$x.r $S3 [OK] -- OUT: $S3 . [OK]
STOP
EOF
test_cmp_count expected.log rot13-filter.log &&
rm -f test2.r "testsubdir/test3 '\''sq'\'',\$x.r" &&
filter_git checkout --quiet --no-progress . &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]
IN: smudge testsubdir/test3 '\''sq'\'',\$x.r $S3 [OK] -- OUT: $S3 . [OK]
STOP
EOF
test_cmp_exclude_clean expected.log rot13-filter.log &&
filter_git checkout --quiet --no-progress empty-branch &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: clean test.r $S [OK] -- OUT: $S . [OK]
STOP
EOF
test_cmp_exclude_clean expected.log rot13-filter.log &&
filter_git checkout --quiet --no-progress master &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: smudge test.r $S [OK] -- OUT: $S . [OK]
IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]
IN: smudge test4-empty.r 0 [OK] -- OUT: 0 [OK]
IN: smudge testsubdir/test3 '\''sq'\'',\$x.r $S3 [OK] -- OUT: $S3 . [OK]
STOP
EOF
test_cmp_exclude_clean expected.log rot13-filter.log &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.r &&
test_cmp_committed_rot13 "$TEST_ROOT/test2.o" test2.r &&
test_cmp_committed_rot13 "$TEST_ROOT/test3 '\''sq'\'',\$x.o" "testsubdir/test3 '\''sq'\'',\$x.r"
)
'
test_expect_success PERL 'required process filter takes precedence' '
test_config_global filter.protocol.clean false &&
test_config_global filter.protocol.process "$TEST_DIRECTORY/t0021/rot13-filter.pl clean" &&
test_config_global filter.protocol.required true &&
rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "*.r filter=protocol" >.gitattributes &&
cp "$TEST_ROOT/test.o" test.r &&
S=$(file_size test.r) &&
# Check that the process filter is invoked here
filter_git add . &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: clean test.r $S [OK] -- OUT: $S . [OK]
STOP
EOF
test_cmp_count expected.log rot13-filter.log
)
'
test_expect_success PERL 'required process filter should be used only for "clean" operation only' '
test_config_global filter.protocol.process "$TEST_DIRECTORY/t0021/rot13-filter.pl clean" &&
rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "*.r filter=protocol" >.gitattributes &&
cp "$TEST_ROOT/test.o" test.r &&
S=$(file_size test.r) &&
filter_git add . &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: clean test.r $S [OK] -- OUT: $S . [OK]
STOP
EOF
test_cmp_count expected.log rot13-filter.log &&
rm test.r &&
filter_git checkout --quiet --no-progress . &&
# If the filter would be used for "smudge", too, we would see
# "IN: smudge test.r 57 [OK] -- OUT: 57 . [OK]" here
cat >expected.log <<-EOF &&
START
init handshake complete
STOP
EOF
test_cmp_exclude_clean expected.log rot13-filter.log
)
'
test_expect_success PERL 'required process filter should process multiple packets' '
test_config_global filter.protocol.process "$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge" &&
test_config_global filter.protocol.required true &&
rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
# Generate data requiring 1, 2, 3 packets
S=65516 && # PKTLINE_DATA_MAXLEN -> Maximal size of a packet
generate_random_characters $(($S )) 1pkt_1__.file &&
generate_random_characters $(($S +1)) 2pkt_1+1.file &&
generate_random_characters $(($S*2-1)) 2pkt_2-1.file &&
generate_random_characters $(($S*2 )) 2pkt_2__.file &&
generate_random_characters $(($S*2+1)) 3pkt_2+1.file &&
for FILE in "$TEST_ROOT"/*.file
do
cp "$FILE" . &&
"$TEST_ROOT/rot13.sh" <"$FILE" >"$FILE.rot13"
done &&
echo "*.file filter=protocol" >.gitattributes &&
filter_git add *.file .gitattributes &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: clean 1pkt_1__.file $(($S )) [OK] -- OUT: $(($S )) . [OK]
IN: clean 2pkt_1+1.file $(($S +1)) [OK] -- OUT: $(($S +1)) .. [OK]
IN: clean 2pkt_2-1.file $(($S*2-1)) [OK] -- OUT: $(($S*2-1)) .. [OK]
IN: clean 2pkt_2__.file $(($S*2 )) [OK] -- OUT: $(($S*2 )) .. [OK]
IN: clean 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK]
STOP
EOF
test_cmp_count expected.log rot13-filter.log &&
rm -f *.file &&
filter_git checkout --quiet --no-progress -- *.file &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: smudge 1pkt_1__.file $(($S )) [OK] -- OUT: $(($S )) . [OK]
IN: smudge 2pkt_1+1.file $(($S +1)) [OK] -- OUT: $(($S +1)) .. [OK]
IN: smudge 2pkt_2-1.file $(($S*2-1)) [OK] -- OUT: $(($S*2-1)) .. [OK]
IN: smudge 2pkt_2__.file $(($S*2 )) [OK] -- OUT: $(($S*2 )) .. [OK]
IN: smudge 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK]
STOP
EOF
test_cmp_exclude_clean expected.log rot13-filter.log &&
for FILE in *.file
do
test_cmp_committed_rot13 "$TEST_ROOT/$FILE" $FILE
done
)
'
test_expect_success PERL 'required process filter with clean error should fail' '
test_config_global filter.protocol.process "$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge" &&
test_config_global filter.protocol.required true &&
rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "*.r filter=protocol" >.gitattributes &&
cp "$TEST_ROOT/test.o" test.r &&
echo "this is going to fail" >clean-write-fail.r &&
echo "content-test3-subdir" >test3.r &&
test_must_fail git add .
)
'
test_expect_success PERL 'process filter should restart after unexpected write failure' '
test_config_global filter.protocol.process "$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge" &&
rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "*.r filter=protocol" >.gitattributes &&
cp "$TEST_ROOT/test.o" test.r &&
cp "$TEST_ROOT/test2.o" test2.r &&
echo "this is going to fail" >smudge-write-fail.o &&
cp smudge-write-fail.o smudge-write-fail.r &&
S=$(file_size test.r) &&
S2=$(file_size test2.r) &&
SF=$(file_size smudge-write-fail.r) &&
git add . &&
rm -f *.r &&
rm -f rot13-filter.log &&
git checkout --quiet --no-progress . 2>git-stderr.log &&
grep "smudge write error at" git-stderr.log &&
grep "error: external filter" git-stderr.log &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: smudge smudge-write-fail.r $SF [OK] -- OUT: $SF [WRITE FAIL]
START
init handshake complete
IN: smudge test.r $S [OK] -- OUT: $S . [OK]
IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]
STOP
EOF
test_cmp_exclude_clean expected.log rot13-filter.log &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.r &&
test_cmp_committed_rot13 "$TEST_ROOT/test2.o" test2.r &&
# Smudge failed
! test_cmp smudge-write-fail.o smudge-write-fail.r &&
"$TEST_ROOT/rot13.sh" <smudge-write-fail.o >expected &&
git cat-file blob :smudge-write-fail.r >actual &&
test_cmp expected actual
)
'
test_expect_success PERL 'process filter should not be restarted if it signals an error' '
test_config_global filter.protocol.process "$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge" &&
rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "*.r filter=protocol" >.gitattributes &&
cp "$TEST_ROOT/test.o" test.r &&
cp "$TEST_ROOT/test2.o" test2.r &&
echo "this will cause an error" >error.o &&
cp error.o error.r &&
S=$(file_size test.r) &&
S2=$(file_size test2.r) &&
SE=$(file_size error.r) &&
git add . &&
rm -f *.r &&
filter_git checkout --quiet --no-progress . &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: smudge error.r $SE [OK] -- OUT: 0 [ERROR]
IN: smudge test.r $S [OK] -- OUT: $S . [OK]
IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]
STOP
EOF
test_cmp_exclude_clean expected.log rot13-filter.log &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.r &&
test_cmp_committed_rot13 "$TEST_ROOT/test2.o" test2.r &&
test_cmp error.o error.r
)
'
test_expect_success PERL 'process filter abort stops processing of all further files' '
test_config_global filter.protocol.process "$TEST_DIRECTORY/t0021/rot13-filter.pl clean smudge" &&
rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "*.r filter=protocol" >.gitattributes &&
cp "$TEST_ROOT/test.o" test.r &&
cp "$TEST_ROOT/test2.o" test2.r &&
echo "error this blob and all future blobs" >abort.o &&
cp abort.o abort.r &&
SA=$(file_size abort.r) &&
git add . &&
rm -f *.r &&
# Note: This test assumes that Git filters files in alphabetical
# order ("abort.r" before "test.r").
filter_git checkout --quiet --no-progress . &&
cat >expected.log <<-EOF &&
START
init handshake complete
IN: smudge abort.r $SA [OK] -- OUT: 0 [ABORT]
STOP
EOF
test_cmp_exclude_clean expected.log rot13-filter.log &&
test_cmp "$TEST_ROOT/test.o" test.r &&
test_cmp "$TEST_ROOT/test2.o" test2.r &&
test_cmp abort.o abort.r
)
'
test_expect_success PERL 'invalid process filter must fail (and not hang!)' '
test_config_global filter.protocol.process cat &&
test_config_global filter.protocol.required true &&
rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "*.r filter=protocol" >.gitattributes &&
cp "$TEST_ROOT/test.o" test.r &&
test_must_fail git add . 2>git-stderr.log &&
grep "does not support filter protocol version" git-stderr.log
)
'
test_done

192
t/t0021/rot13-filter.pl Executable file
Просмотреть файл

@ -0,0 +1,192 @@
#!/usr/bin/perl
#
# Example implementation for the Git filter protocol version 2
# See Documentation/gitattributes.txt, section "Filter Protocol"
#
# The script takes the list of supported protocol capabilities as
# arguments ("clean", "smudge", etc).
#
# This implementation supports special test cases:
# (1) If data with the pathname "clean-write-fail.r" is processed with
# a "clean" operation then the write operation will die.
# (2) If data with the pathname "smudge-write-fail.r" is processed with
# a "smudge" operation then the write operation will die.
# (3) If data with the pathname "error.r" is processed with any
# operation then the filter signals that it cannot or does not want
# to process the file.
# (4) If data with the pathname "abort.r" is processed with any
# operation then the filter signals that it cannot or does not want
# to process the file and any file after that is processed with the
# same command.
#
use strict;
use warnings;
my $MAX_PACKET_CONTENT_SIZE = 65516;
my @capabilities = @ARGV;
open my $debug, ">>", "rot13-filter.log" or die "cannot open log file: $!";
sub rot13 {
my $str = shift;
$str =~ y/A-Za-z/N-ZA-Mn-za-m/;
return $str;
}
sub packet_bin_read {
my $buffer;
my $bytes_read = read STDIN, $buffer, 4;
if ( $bytes_read == 0 ) {
# EOF - Git stopped talking to us!
print $debug "STOP\n";
exit();
}
elsif ( $bytes_read != 4 ) {
die "invalid packet: '$buffer'";
}
my $pkt_size = hex($buffer);
if ( $pkt_size == 0 ) {
return ( 1, "" );
}
elsif ( $pkt_size > 4 ) {
my $content_size = $pkt_size - 4;
$bytes_read = read STDIN, $buffer, $content_size;
if ( $bytes_read != $content_size ) {
die "invalid packet ($content_size bytes expected; $bytes_read bytes read)";
}
return ( 0, $buffer );
}
else {
die "invalid packet size: $pkt_size";
}
}
sub packet_txt_read {
my ( $res, $buf ) = packet_bin_read();
unless ( $buf =~ s/\n$// ) {
die "A non-binary line MUST be terminated by an LF.";
}
return ( $res, $buf );
}
sub packet_bin_write {
my $buf = shift;
print STDOUT sprintf( "%04x", length($buf) + 4 );
print STDOUT $buf;
STDOUT->flush();
}
sub packet_txt_write {
packet_bin_write( $_[0] . "\n" );
}
sub packet_flush {
print STDOUT sprintf( "%04x", 0 );
STDOUT->flush();
}
print $debug "START\n";
$debug->flush();
( packet_txt_read() eq ( 0, "git-filter-client" ) ) || die "bad initialize";
( packet_txt_read() eq ( 0, "version=2" ) ) || die "bad version";
( packet_bin_read() eq ( 1, "" ) ) || die "bad version end";
packet_txt_write("git-filter-server");
packet_txt_write("version=2");
packet_flush();
( packet_txt_read() eq ( 0, "capability=clean" ) ) || die "bad capability";
( packet_txt_read() eq ( 0, "capability=smudge" ) ) || die "bad capability";
( packet_bin_read() eq ( 1, "" ) ) || die "bad capability end";
foreach (@capabilities) {
packet_txt_write( "capability=" . $_ );
}
packet_flush();
print $debug "init handshake complete\n";
$debug->flush();
while (1) {
my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;
print $debug "IN: $command";
$debug->flush();
my ($pathname) = packet_txt_read() =~ /^pathname=([^=]+)$/;
print $debug " $pathname";
$debug->flush();
# Flush
packet_bin_read();
my $input = "";
{
binmode(STDIN);
my $buffer;
my $done = 0;
while ( !$done ) {
( $done, $buffer ) = packet_bin_read();
$input .= $buffer;
}
print $debug " " . length($input) . " [OK] -- ";
$debug->flush();
}
my $output;
if ( $pathname eq "error.r" or $pathname eq "abort.r" ) {
$output = "";
}
elsif ( $command eq "clean" and grep( /^clean$/, @capabilities ) ) {
$output = rot13($input);
}
elsif ( $command eq "smudge" and grep( /^smudge$/, @capabilities ) ) {
$output = rot13($input);
}
else {
die "bad command '$command'";
}
print $debug "OUT: " . length($output) . " ";
$debug->flush();
if ( $pathname eq "error.r" ) {
print $debug "[ERROR]\n";
$debug->flush();
packet_txt_write("status=error");
packet_flush();
}
elsif ( $pathname eq "abort.r" ) {
print $debug "[ABORT]\n";
$debug->flush();
packet_txt_write("status=abort");
packet_flush();
}
else {
packet_txt_write("status=success");
packet_flush();
if ( $pathname eq "${command}-write-fail.r" ) {
print $debug "[WRITE FAIL]\n";
$debug->flush();
die "${command} write error";
}
while ( length($output) > 0 ) {
my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );
packet_bin_write($packet);
# dots represent the number of packets
print $debug ".";
if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {
$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );
}
else {
$output = "";
}
}
packet_flush();
print $debug " [OK]\n";
$debug->flush();
packet_flush();
}
}

Просмотреть файл

@ -398,13 +398,13 @@ static int get_common_commits(void)
if (multi_ack == 2 && got_common
&& !got_other && ok_to_give_up()) {
sent_ready = 1;
packet_write(1, "ACK %s ready\n", last_hex);
packet_write_fmt(1, "ACK %s ready\n", last_hex);
}
if (have_obj.nr == 0 || multi_ack)
packet_write(1, "NAK\n");
packet_write_fmt(1, "NAK\n");
if (no_done && sent_ready) {
packet_write(1, "ACK %s\n", last_hex);
packet_write_fmt(1, "ACK %s\n", last_hex);
return 0;
}
if (stateless_rpc)
@ -421,20 +421,20 @@ static int get_common_commits(void)
const char *hex = sha1_to_hex(sha1);
if (multi_ack == 2) {
sent_ready = 1;
packet_write(1, "ACK %s ready\n", hex);
packet_write_fmt(1, "ACK %s ready\n", hex);
} else
packet_write(1, "ACK %s continue\n", hex);
packet_write_fmt(1, "ACK %s continue\n", hex);
}
break;
default:
got_common = 1;
memcpy(last_hex, sha1_to_hex(sha1), 41);
if (multi_ack == 2)
packet_write(1, "ACK %s common\n", last_hex);
packet_write_fmt(1, "ACK %s common\n", last_hex);
else if (multi_ack)
packet_write(1, "ACK %s continue\n", last_hex);
packet_write_fmt(1, "ACK %s continue\n", last_hex);
else if (have_obj.nr == 1)
packet_write(1, "ACK %s\n", last_hex);
packet_write_fmt(1, "ACK %s\n", last_hex);
break;
}
continue;
@ -442,10 +442,10 @@ static int get_common_commits(void)
if (!strcmp(line, "done")) {
if (have_obj.nr > 0) {
if (multi_ack)
packet_write(1, "ACK %s\n", last_hex);
packet_write_fmt(1, "ACK %s\n", last_hex);
return 0;
}
packet_write(1, "NAK\n");
packet_write_fmt(1, "NAK\n");
return -1;
}
die("git upload-pack: expected SHA1 list, got '%s'", line);
@ -638,8 +638,8 @@ static void send_shallow(struct commit_list *result)
while (result) {
struct object *object = &result->item->object;
if (!(object->flags & (CLIENT_SHALLOW|NOT_SHALLOW))) {
packet_write(1, "shallow %s",
oid_to_hex(&object->oid));
packet_write_fmt(1, "shallow %s",
oid_to_hex(&object->oid));
register_shallow(object->oid.hash);
shallow_nr++;
}
@ -655,8 +655,8 @@ static void send_unshallow(const struct object_array *shallows)
struct object *object = shallows->objects[i].item;
if (object->flags & NOT_SHALLOW) {
struct commit_list *parents;
packet_write(1, "unshallow %s",
oid_to_hex(&object->oid));
packet_write_fmt(1, "unshallow %s",
oid_to_hex(&object->oid));
object->flags &= ~CLIENT_SHALLOW;
/*
* We want to _register_ "object" as shallow, but we
@ -932,7 +932,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
struct strbuf symref_info = STRBUF_INIT;
format_symref_info(&symref_info, cb_data);
packet_write(1, "%s %s%c%s%s%s%s%s agent=%s\n",
packet_write_fmt(1, "%s %s%c%s%s%s%s%s agent=%s\n",
oid_to_hex(oid), refname_nons,
0, capabilities,
(allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?
@ -944,11 +944,11 @@ static int send_ref(const char *refname, const struct object_id *oid,
git_user_agent_sanitized());
strbuf_release(&symref_info);
} else {
packet_write(1, "%s %s\n", oid_to_hex(oid), refname_nons);
packet_write_fmt(1, "%s %s\n", oid_to_hex(oid), refname_nons);
}
capabilities = NULL;
if (!peel_ref(refname, peeled.hash))
packet_write(1, "%s %s^{}\n", oid_to_hex(&peeled), refname_nons);
packet_write_fmt(1, "%s %s^{}\n", oid_to_hex(&peeled), refname_nons);
return 0;
}

Просмотреть файл

@ -1,19 +1,6 @@
#include "cache.h"
#include "run-command.h"
static void check_pipe(int err)
{
if (err == EPIPE) {
if (in_async())
async_exit(141);
signal(SIGPIPE, SIG_DFL);
raise(SIGPIPE);
/* Should never happen, but just in case... */
exit(141);
}
}
/*
* Some cases use stdio, but want to flush after the write
* to get error handling (and to get better interactive