Граф коммитов

23 Коммитов

Автор SHA1 Сообщение Дата
Junio C Hamano 324b6b1678 Merge branch 'mh/check-attr-relative'
* mh/check-attr-relative: (29 commits)
  test-path-utils: Add subcommand "prefix_path"
  test-path-utils: Add subcommand "absolute_path"
  git-check-attr: Normalize paths
  git-check-attr: Demonstrate problems with relative paths
  git-check-attr: Demonstrate problems with unnormalized paths
  git-check-attr: test that no output is written to stderr
  Rename git_checkattr() to git_check_attr()
  git-check-attr: Fix command-line handling to match docs
  git-check-attr: Drive two tests using the same raw data
  git-check-attr: Add an --all option to show all attributes
  git-check-attr: Error out if no pathnames are specified
  git-check-attr: Process command-line args more systematically
  git-check-attr: Handle each error separately
  git-check-attr: Extract a function error_with_usage()
  git-check-attr: Introduce a new variable
  git-check-attr: Extract a function output_attr()
  Allow querying all attributes on a file
  Remove redundant check
  Remove redundant call to bootstrap_attr_stack()
  Extract a function collect_all_attrs()
  ...
2011-08-17 17:36:22 -07:00
Junio C Hamano 96790ca029 Merge branch 'jc/pack-order-tweak'
* jc/pack-order-tweak:
  pack-objects: optimize "recency order"
  core: log offset pack data accesses happened
2011-08-05 14:54:57 -07:00
Michael Haggerty d932f4eb9f Rename git_checkattr() to git_check_attr()
Suggested by: Junio Hamano <gitster@pobox.com>

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-08-04 15:53:21 -07:00
Junio C Hamano d907bf8ef3 Merge branch 'jc/index-pack'
* jc/index-pack:
  verify-pack: use index-pack --verify
  index-pack: show histogram when emulating "verify-pack -v"
  index-pack: start learning to emulate "verify-pack -v"
  index-pack: a miniscule refactor
  index-pack --verify: read anomalous offsets from v2 idx file
  write_idx_file: need_large_offset() helper function
  index-pack: --verify
  write_idx_file: introduce a struct to hold idx customization options
  index-pack: group the delta-base array entries also by type

Conflicts:
	builtin/verify-pack.c
	cache.h
	sha1_file.c
2011-07-19 09:54:51 -07:00
Junio C Hamano 1b4bb16b9e pack-objects: optimize "recency order"
This optimizes the "recency order" (see pack-heuristics.txt in
Documentation/technical/ directory) used to order objects within a
packfile in three ways:

 - Commits at the tip of tags are written together, in the hope that
   revision traversal done in incremental fetch (which starts by
   putting them in a revision queue marked as UNINTERESTING) will see a
   better locality of these objects;

 - In the original recency order, trees and blobs are intermixed. Write
   trees together before blobs, in the hope that this will improve
   locality when running pathspec-limited revision traversal, i.e.
   "git log paths...";

 - When writing blob objects out, write the whole family of blobs that use
   the same delta base object together, by starting from the root of the
   delta chain, and writing its immediate children in a width-first
   manner, in the hope that this will again improve locality when reading
   blobs that belong to the same path, which are likely to be deltified
   against each other.

I tried various workloads in the Linux kernel repositories (HEAD at
v3.0-rc6-71-g4dd1b49) packed with v1.7.6 and with this patch, counting how
large seeks are needed between adjacent accesses to objects in the pack,
and the result looks promising.  The history has 2072052 objects, weighing
some 490MiB.

 * Simple commit-only log.

   $ git log >/dev/null

   There are 254656 commits in total.

                                  v1.7.6  with patch
   Total number of access :      258,031     258,032
          0.0% percentile :           12          12
         10.0% percentile :          259         259
         20.0% percentile :          294         294
         30.0% percentile :          326         326
         40.0% percentile :          363         363
         50.0% percentile :          415         415
         60.0% percentile :          513         513
         70.0% percentile :          857         858
         80.0% percentile :       10,434      10,441
         90.0% percentile :       91,985      91,996
         95.0% percentile :      260,852     260,885
         99.0% percentile :    1,150,680   1,152,811
         99.9% percentile :    3,148,435   3,148,435
       Less than 2MiB seek:       99.70%      99.69%

   95% of the pack accesses look at data that is no further than 260kB
   from the previous location we accessed. The patch does not change the
   order of commit objects very much, and the result is very similar.

 * Pathspec-limited log.

   $ git log drivers/net >/dev/null

   The path is touched by 26551 commits and merges (among 254656 total).

                                  v1.7.6  with patch
   Total number of access :      559,511     558,663
          0.0% percentile :            0           0
         10.0% percentile :          182         167
         20.0% percentile :          259         233
         30.0% percentile :          357         304
         40.0% percentile :          714         485
         50.0% percentile :        5,046       3,976
         60.0% percentile :      688,671     443,578
         70.0% percentile :  319,574,732 110,370,100
         80.0% percentile :  361,647,599 123,707,229
         90.0% percentile :  393,195,669 128,947,636
         95.0% percentile :  405,496,875 131,609,321
         99.0% percentile :  412,942,470 133,078,115
         99.5% percentile :  413,172,266 133,163,349
         99.9% percentile :  413,354,356 133,240,445
       Less than 2MiB seek:       61.71%      62.87%

   With the current pack heuristics, more than 30% of accesses have to
   seek further than 300MB; the updated pack heuristics ensures that less
   than 0.1% of accesses have to seek further than 135MB. This is largely
   due to the fact that the updated heuristics does not mix blobs and
   trees together.

 * Blame.

   $ git blame drivers/net/ne.c >/dev/null

   The path is touched by 34 commits and merges.

                                  v1.7.6  with patch
   Total number of access :      178,147     178,166
          0.0% percentile :            0           0
         10.0% percentile :          142         139
         20.0% percentile :          222         194
         30.0% percentile :          373         300
         40.0% percentile :        1,168         837
         50.0% percentile :       11,248       7,334
         60.0% percentile :  305,121,284 106,850,130
         70.0% percentile :  361,427,854 123,709,715
         80.0% percentile :  388,127,343 128,171,047
         90.0% percentile :  399,987,762 130,200,707
         95.0% percentile :  408,230,673 132,174,308
         99.0% percentile :  412,947,017 133,181,160
         99.5% percentile :  413,312,798 133,220,425
         99.9% percentile :  413,352,366 133,269,051
       Less than 2MiB seek:       56.47%      56.83%

   The result is very similar to the pathspec-limited log above, which
   only looks at the tree objects.

 * Packing recent history.

   $ (git for-each-ref --format='^%(refname)' refs/tags; echo HEAD) |
     git pack-objects --revs --stdout >/dev/null

   This should pack data worth 71 commits.

                                  v1.7.6  with patch
   Total number of access :       11,511      11,514
          0.0% percentile :            0           0
         10.0% percentile :           48          47
         20.0% percentile :          134          98
         30.0% percentile :          332         178
         40.0% percentile :        1,386         293
         50.0% percentile :        8,030         478
         60.0% percentile :       33,676       1,195
         70.0% percentile :      147,268      26,216
         80.0% percentile :    9,178,662     464,598
         90.0% percentile :   67,922,665     965,782
         95.0% percentile :   87,773,251   1,226,102
         99.0% percentile :   98,011,763   1,932,377
         99.5% percentile :  100,074,427  33,642,128
         99.9% percentile :  105,336,398 275,772,650
       Less than 2MiB seek:       77.09%      99.04%

    The long-tail part of the result looks worse with the patch, but
    the change helps majority of the access. 99.04% of the accesses
    need less than 2MiB of seeking, compared to 77.09% with the current
    packing heuristics.

 * Index pack.

   $ git index-pack -v .git/objects/pack/pack*.pack

                                  v1.7.6  with patch
   Total number of access :    2,791,228   2,788,802
          0.0% percentile :            9           9
         10.0% percentile :          140          89
         20.0% percentile :          233         167
         30.0% percentile :          322         235
         40.0% percentile :          464         310
         50.0% percentile :          862         423
         60.0% percentile :        2,566         686
         70.0% percentile :       25,827       1,498
         80.0% percentile :    1,317,862       4,971
         90.0% percentile :   11,926,385     119,398
         95.0% percentile :   41,304,149     952,519
         99.0% percentile :  227,613,070   6,709,650
         99.5% percentile :  321,265,121  11,734,871
         99.9% percentile :  382,919,785  33,155,191
       Less than 2MiB seek:       81.73%      96.92%

   As the index-pack command already walks objects in the delta chain
   order, writing the blobs out in the delta chain order seems to
   drastically improve the locality of access.

Note that a half-a-gigabyte packfile comfortably fits in the buffer cache,
and you would unlikely to see much performance difference on a modern and
reasonably beefy machine with enough memory and local disks. Benchmarking
with cold cache (or over NFS) would be interesting.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-07-08 10:03:24 -07:00
Junio C Hamano ef49a7a012 zlib: zlib can only process 4GB at a time
The size of objects we read from the repository and data we try to put
into the repository are represented in "unsigned long", so that on larger
architectures we can handle objects that weigh more than 4GB.

But the interface defined in zlib.h to communicate with inflate/deflate
limits avail_in (how many bytes of input are we calling zlib with) and
avail_out (how many bytes of output from zlib are we ready to accept)
fields effectively to 4GB by defining their type to be uInt.

In many places in our code, we allocate a large buffer (e.g. mmap'ing a
large loose object file) and tell zlib its size by assigning the size to
avail_in field of the stream, but that will truncate the high octets of
the real size. The worst part of this story is that we often pass around
z_stream (the state object used by zlib) to keep track of the number of
used bytes in input/output buffer by inspecting these two fields, which
practically limits our callchain to the same 4GB limit.

Wrap z_stream in another structure git_zstream that can express avail_in
and avail_out in unsigned long. For now, just die() when the caller gives
a size that cannot be given to a single zlib call. In later patches in the
series, we would make git_inflate() and git_deflate() internally loop to
give callers an illusion that our "improved" version of zlib interface can
operate on a buffer larger than 4GB in one go.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-06-10 11:52:15 -07:00
Junio C Hamano 225a6f1068 zlib: wrap deflateBound() too
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-06-10 11:18:17 -07:00
Junio C Hamano 55bb5c9147 zlib: wrap deflate side of the API
Wrap deflateInit, deflate, and deflateEnd for everybody, and the sole use
of deflateInit2 in remote-curl.c to tell the library to use gzip header
and trailer in git_deflate_init_gzip().

There is only one caller that cares about the status from deflateEnd().
Introduce git_deflate_end_gently() to let that sole caller retrieve the
status and act on it (i.e. die) for now, but we would probably want to
make inflate_end/deflate_end die when they ran out of memory and get
rid of the _gently() kind.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-06-10 11:10:29 -07:00
Junio C Hamano 15366280c2 Teach core.bigfilethreashold to pack-objects
The pack-objects command should take notice of the object file and
refrain from attempting to delta large ones, to be consistent with
the fast-import command.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-04-05 20:25:49 -07:00
Junio C Hamano ebcfb3791a write_idx_file: introduce a struct to hold idx customization options
Remove two globals, pack_idx_default version and pack_idx_off32_limit,
and place them in a pack_idx_option structure.  Allow callers to pass
it to write_idx_file() as a parameter.

Adjust all callers to the API change.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-27 23:29:03 -08:00
Junio C Hamano b361888dd5 thread-utils.h: simplify the inclusion
All files that include this header file use the same four line
incantation:

    #ifndef NO_PTHREADS
    #include <pthread.h>
    #include "thread-utils.h"
    #endif

Move the responsibility for that gymnastics to the header file from the
files that include it.  This approach makes it easier to later declare new
services that are related to threading in thread-utils.h and have them
available to all the threading code.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-10 12:58:06 -08:00
Junio C Hamano 39f04dbaac Merge branch 'jn/thinner-wrapper'
* jn/thinner-wrapper:
  Remove pack file handling dependency from wrapper.o
  pack-objects: mark file-local variable static
  wrapper: give zlib wrappers their own translation unit
  strbuf: move strbuf_branchname to sha1_name.c
  path helpers: move git_mkstemp* to wrapper.c
  wrapper: move odb_* to environment.c
  wrapper: move xmmap() to sha1_file.c
2010-12-03 16:13:06 -08:00
Jonathan Nieder bc9b21755e pack-objects: mark file-local variable static
old_try_to_free_routine is not meant for use from other files.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-11-10 11:08:04 -08:00
Nicolas Pitre 71064a956b make pack-objects a bit more resilient to repo corruption
Right now, packing valid objects could fail when creating a thin pack
simply because a pack edge object used as a preferred base is corrupted.
Since preferred base objects are not strictly needed to produce a valid
pack, let's not consider the inability to read them as a fatal error.
Delta compression may well be attempted against other objects in the
search window.  To avoid warning storms (we are in the inner loop of
the delta search window) a warning is emitted only on the first
occurrence.

Signed-off-by: Nicolas Pitre <nico@fluxnic.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-22 14:59:58 -07:00
Štěpán Němec 884220653f Put a space between `<' and argument in pack-objects usage string
This makes it cosistent with other places (including the
git-pack-objects(1) manpage itself) and avoids possible confusion (I,
for one, mistook `<object-list' for a `<object-list>' typo at first when
preparing this series).

Signed-off-by: Štěpán Němec <stepnem@gmail.com>
Acked-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-08 12:31:08 -07:00
Štěpán Němec 0adda9362a Use parentheses and `...' where appropriate
Remove some stray usage of other bracket types and asterisks for the
same purpose.

Signed-off-by: Štěpán Němec <stepnem@gmail.com>
Acked-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-08 12:31:07 -07:00
Štěpán Němec 62b4698e55 Use angles for placeholders consistently
Signed-off-by: Štěpán Němec <stepnem@gmail.com>
Acked-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-08 12:29:52 -07:00
Erik Faye-Lund c03c83152d do not depend on signed integer overflow
Signed integer overflow is not defined in C, so do not depend on it.

This fixes a problem with GCC 4.4.0 and -O3 where the optimizer would
consider "consumed_bytes > consumed_bytes + bytes" as a constant
expression, and never execute the die()-call.

Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
Acked-by: Nicolas Pitre <nico@fluxnic.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-06 11:10:07 -07:00
Johannes Schindelin 8695353147 Fix typo in pack-objects' usage
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Pat Thoyts <patthoyts@users.sourceforge.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-09-30 12:22:02 -07:00
Junio C Hamano 44e08b003d Merge branch 'js/try-to-free-stackable'
* js/try-to-free-stackable:
  Do not call release_pack_memory in malloc wrappers when GIT_TRACE is used
  Have set_try_to_free_routine return the previous routine
2010-06-13 11:21:21 -07:00
Junio C Hamano ea5f75a64a Merge branch 'np/malloc-threading'
* np/malloc-threading:
  Thread-safe xmalloc and xrealloc needs a recursive mutex
  Make xmalloc and xrealloc thread-safe
2010-05-21 04:02:16 -07:00
Junio C Hamano 2e0e8b68e3 Merge branch 'lt/deepen-builtin-source'
* lt/deepen-builtin-source:
  Move 'builtin-*' into a 'builtin/' subdirectory

Conflicts:
	Makefile
2010-03-10 15:25:18 -08:00
Linus Torvalds 81b50f3ce4 Move 'builtin-*' into a 'builtin/' subdirectory
This shrinks the top-level directory a bit, and makes it much more
pleasant to use auto-completion on the thing. Instead of

	[torvalds@nehalem git]$ em buil<tab>
	Display all 180 possibilities? (y or n)
	[torvalds@nehalem git]$ em builtin-sh
	builtin-shortlog.c     builtin-show-branch.c  builtin-show-ref.c
	builtin-shortlog.o     builtin-show-branch.o  builtin-show-ref.o
	[torvalds@nehalem git]$ em builtin-shor<tab>
	builtin-shortlog.c  builtin-shortlog.o
	[torvalds@nehalem git]$ em builtin-shortlog.c

you get

	[torvalds@nehalem git]$ em buil<tab>		[type]
	builtin/   builtin.h
	[torvalds@nehalem git]$ em builtin		[auto-completes to]
	[torvalds@nehalem git]$ em builtin/sh<tab>	[type]
	shortlog.c     shortlog.o     show-branch.c  show-branch.o  show-ref.c     show-ref.o
	[torvalds@nehalem git]$ em builtin/sho		[auto-completes to]
	[torvalds@nehalem git]$ em builtin/shor<tab>	[type]
	shortlog.c  shortlog.o
	[torvalds@nehalem git]$ em builtin/shortlog.c

which doesn't seem all that different, but not having that annoying
break in "Display all 180 possibilities?" is quite a relief.

NOTE! If you do this in a clean tree (no object files etc), or using an
editor that has auto-completion rules that ignores '*.o' files, you
won't see that annoying 'Display all 180 possibilities?' message - it
will just show the choices instead.  I think bash has some cut-off
around 100 choices or something.

So the reason I see this is that I'm using an odd editory, and thus
don't have the rules to cut down on auto-completion.  But you can
simulate that by using 'ls' instead, or something similar.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-02-22 14:29:41 -08:00