Skip to content

Commit 0649303

Browse files
gitstergitster
committed
Merge branch 'tb/multi-pack-bitmaps'
The reachability bitmap file used to be generated only for a single pack, but now we've learned to generate bitmaps for history that span across multiple packfiles. * tb/multi-pack-bitmaps: (29 commits) pack-bitmap: drop bitmap_index argument from try_partial_reuse() pack-bitmap: drop repository argument from prepare_midx_bitmap_git() p5326: perf tests for MIDX bitmaps p5310: extract full and partial bitmap tests midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' t7700: update to work with MIDX bitmap test knob t5319: don't write MIDX bitmaps in t5319 t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP t0410: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP t5326: test multi-pack bitmap behavior t/helper/test-read-midx.c: add --checksum mode t5310: move some tests to lib-bitmap.sh pack-bitmap: write multi-pack bitmaps pack-bitmap: read multi-pack bitmaps pack-bitmap.c: avoid redundant calls to try_partial_reuse pack-bitmap.c: introduce 'bitmap_is_preferred_refname()' pack-bitmap.c: introduce 'nth_bitmap_object_oid()' pack-bitmap.c: introduce 'bitmap_num_objects()' midx: avoid opening multiple MIDXs when writing midx: close linked MIDXs, avoid leaking memory ...
2 parents deec8aa + 73cd7d9 commit 0649303

28 files changed

+1676
-465
lines changed

‎Documentation/git-multi-pack-index.txt‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ SYNOPSIS
1010
--------
1111
[verse]
1212
'git multi-pack-index' [--object-dir=<dir>] [--[no-]progress]
13-
[--preferred-pack=<pack>] <subcommand>
13+
[--preferred-pack=<pack>] [--[no-]bitmap] <subcommand>
1414

1515
DESCRIPTION
1616
-----------
@@ -23,6 +23,8 @@ OPTIONS
2323
Use given directory for the location of Git objects. We check
2424
`<dir>/packs/multi-pack-index` for the current MIDX file, and
2525
`<dir>/packs` for the pack-files to index.
26+
+
27+
`<dir>` must be an alternate of the current repository.
2628

2729
--[no-]progress::
2830
Turn progress on/off explicitly. If neither is specified, progress is
@@ -37,9 +39,12 @@ write::
3739
--
3840
--preferred-pack=<pack>::
3941
Optionally specify the tie-breaking pack used when
40-
multiple packs contain the same object. If not given,
41-
ties are broken in favor of the pack with the lowest
42-
mtime.
42+
multiple packs contain the same object. `<pack>` must
43+
contain at least one object. If not given, ties are
44+
broken in favor of the pack with the lowest mtime.
45+
46+
--[no-]bitmap::
47+
Control whether or not a multi-pack bitmap is written.
4348
--
4449

4550
verify::
@@ -81,6 +86,13 @@ EXAMPLES
8186
$ git multi-pack-index write
8287
-----------------------------------------------
8388

89+
* Write a MIDX file for the packfiles in the current .git folder with a
90+
corresponding bitmap.
91+
+
92+
-------------------------------------------------------------
93+
$ git multi-pack-index write --preferred-pack=<pack> --bitmap
94+
-------------------------------------------------------------
95+
8496
* Write a MIDX file for the packfiles in an alternate object store.
8597
+
8698
-----------------------------------------------

‎Documentation/technical/bitmap-format.txt‎

Lines changed: 57 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,44 @@
11
GIT bitmap v1 format
22
====================
33

4+
== Pack and multi-pack bitmaps
5+
6+
Bitmaps store reachability information about the set of objects in a packfile,
7+
or a multi-pack index (MIDX). The former is defined obviously, and the latter is
8+
defined as the union of objects in packs contained in the MIDX.
9+
10+
A bitmap may belong to either one pack, or the repository's multi-pack index (if
11+
it exists). A repository may have at most one bitmap.
12+
13+
An object is uniquely described by its bit position within a bitmap:
14+
15+
- If the bitmap belongs to a packfile, the __n__th bit corresponds to
16+
the __n__th object in pack order. For a function `offset` which maps
17+
objects to their byte offset within a pack, pack order is defined as
18+
follows:
19+
20+
o1 <= o2 <==> offset(o1) <= offset(o2)
21+
22+
- If the bitmap belongs to a MIDX, the __n__th bit corresponds to the
23+
__n__th object in MIDX order. With an additional function `pack` which
24+
maps objects to the pack they were selected from by the MIDX, MIDX order
25+
is defined as follows:
26+
27+
o1 <= o2 <==> pack(o1) <= pack(o2) /\ offset(o1) <= offset(o2)
28+
29+
The ordering between packs is done according to the MIDX's .rev file.
30+
Notably, the preferred pack sorts ahead of all other packs.
31+
32+
The on-disk representation (described below) of a bitmap is the same regardless
33+
of whether or not that bitmap belongs to a packfile or a MIDX. The only
34+
difference is the interpretation of the bits, which is described above.
35+
36+
Certain bitmap extensions are supported (see: Appendix B). No extensions are
37+
required for bitmaps corresponding to packfiles. For bitmaps that correspond to
38+
MIDXs, both the bit-cache and rev-cache extensions are required.
39+
40+
== On-disk format
41+
442
- A header appears at the beginning:
543

644
4-byte signature: {'B', 'I', 'T', 'M'}
@@ -14,17 +52,19 @@ GIT bitmap v1 format
1452
The following flags are supported:
1553

1654
- BITMAP_OPT_FULL_DAG (0x1) REQUIRED
17-
This flag must always be present. It implies that the bitmap
18-
index has been generated for a packfile with full closure
19-
(i.e. where every single object in the packfile can find
20-
its parent links inside the same packfile). This is a
21-
requirement for the bitmap index format, also present in JGit,
22-
that greatly reduces the complexity of the implementation.
55+
This flag must always be present. It implies that the
56+
bitmap index has been generated for a packfile or
57+
multi-pack index (MIDX) with full closure (i.e. where
58+
every single object in the packfile/MIDX can find its
59+
parent links inside the same packfile/MIDX). This is a
60+
requirement for the bitmap index format, also present in
61+
JGit, that greatly reduces the complexity of the
62+
implementation.
2363

2464
- BITMAP_OPT_HASH_CACHE (0x4)
2565
If present, the end of the bitmap file contains
2666
`N` 32-bit name-hash values, one per object in the
27-
pack. The format and meaning of the name-hash is
67+
pack/MIDX. The format and meaning of the name-hash is
2868
described below.
2969

3070
4-byte entry count (network byte order)
@@ -33,7 +73,8 @@ GIT bitmap v1 format
3373

3474
20-byte checksum
3575

36-
The SHA1 checksum of the pack this bitmap index belongs to.
76+
The SHA1 checksum of the pack/MIDX this bitmap index
77+
belongs to.
3778

3879
- 4 EWAH bitmaps that act as type indexes
3980

@@ -50,7 +91,7 @@ GIT bitmap v1 format
5091
- Tags
5192

5293
In each bitmap, the `n`th bit is set to true if the `n`th object
53-
in the packfile is of that type.
94+
in the packfile or multi-pack index is of that type.
5495

5596
The obvious consequence is that the OR of all 4 bitmaps will result
5697
in a full set (all bits set), and the AND of all 4 bitmaps will
@@ -62,8 +103,9 @@ GIT bitmap v1 format
62103
Each entry contains the following:
63104

64105
- 4-byte object position (network byte order)
65-
The position **in the index for the packfile** where the
66-
bitmap for this commit is found.
106+
The position **in the index for the packfile or
107+
multi-pack index** where the bitmap for this commit is
108+
found.
67109

68110
- 1-byte XOR-offset
69111
The xor offset used to compress this bitmap. For an entry
@@ -146,10 +188,11 @@ Name-hash cache
146188
---------------
147189

148190
If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains
149-
a cache of 32-bit values, one per object in the pack. The value at
191+
a cache of 32-bit values, one per object in the pack/MIDX. The value at
150192
position `i` is the hash of the pathname at which the `i`th object
151-
(counting in index order) in the pack can be found. This can be fed
152-
into the delta heuristics to compare objects with similar pathnames.
193+
(counting in index or multi-pack index order) in the pack/MIDX can be found.
194+
This can be fed into the delta heuristics to compare objects with similar
195+
pathnames.
153196

154197
The hash algorithm used is:
155198

‎Documentation/technical/multi-pack-index.txt‎

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -71,14 +71,10 @@ Future Work
7171
still reducing the number of binary searches required for object
7272
lookups.
7373

74-
- The reachability bitmap is currently paired directly with a single
75-
packfile, using the pack-order as the object order to hopefully
76-
compress the bitmaps well using run-length encoding. This could be
77-
extended to pair a reachability bitmap with a multi-pack-index. If
78-
the multi-pack-index is extended to store a "stable object order"
74+
- If the multi-pack-index is extended to store a "stable object order"
7975
(a function Order(hash) = integer that is constant for a given hash,
80-
even as the multi-pack-index is updated) then a reachability bitmap
81-
could point to a multi-pack-index and be updated independently.
76+
even as the multi-pack-index is updated) then MIDX bitmaps could be
77+
updated independently of the MIDX.
8278

8379
- Packfiles can be marked as "special" using empty files that share
8480
the initial name but replace ".pack" with ".keep" or ".promisor".

‎builtin/commit-graph.c‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -60,28 +60,6 @@ static struct option *add_common_options(struct option *to)
6060
return parse_options_concat(common_opts, to);
6161
}
6262

63-
static struct object_directory *find_odb(struct repository *r,
64-
const char *obj_dir)
65-
{
66-
struct object_directory *odb;
67-
char *obj_dir_real = real_pathdup(obj_dir, 1);
68-
struct strbuf odb_path_real = STRBUF_INIT;
69-
70-
prepare_alt_odb(r);
71-
for (odb = r->objects->odb; odb; odb = odb->next) {
72-
strbuf_realpath(&odb_path_real, odb->path, 1);
73-
if (!strcmp(obj_dir_real, odb_path_real.buf))
74-
break;
75-
}
76-
77-
free(obj_dir_real);
78-
strbuf_release(&odb_path_real);
79-
80-
if (!odb)
81-
die(_("could not find object directory matching %s"), obj_dir);
82-
return odb;
83-
}
84-
8563
static int graph_verify(int argc, const char **argv)
8664
{
8765
struct commit_graph *graph = NULL;

‎builtin/multi-pack-index.c‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
6868
OPT_STRING(0, "preferred-pack", &opts.preferred_pack,
6969
N_("preferred-pack"),
7070
N_("pack for reuse when computing a multi-pack bitmap")),
71+
OPT_BIT(0, "bitmap", &opts.flags, N_("write multi-pack bitmap"),
72+
MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),
7173
OPT_END(),
7274
};
7375

‎builtin/pack-objects.c‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1124,6 +1124,11 @@ static void write_reused_pack(struct hashfile *f)
11241124
break;
11251125

11261126
offset += ewah_bit_ctz64(word >> offset);
1127+
/*
1128+
* Can use bit positions directly, even for MIDX
1129+
* bitmaps. See comment in try_partial_reuse()
1130+
* for why.
1131+
*/
11271132
write_reused_pack_one(pos + offset, f, &w_curs);
11281133
display_progress(progress_state, ++written);
11291134
}
@@ -1256,7 +1261,8 @@ static void write_pack_file(void)
12561261

12571262
bitmap_writer_show_progress(progress);
12581263
bitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);
1259-
bitmap_writer_build(&to_pack);
1264+
if (bitmap_writer_build(&to_pack) < 0)
1265+
die(_("failed to write bitmap index"));
12601266
bitmap_writer_finish(written_list, nr_written,
12611267
tmpname.buf, write_bitmap_options);
12621268
write_bitmap_index = 0;

‎builtin/repack.c‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -515,6 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
515515
if (!(pack_everything & ALL_INTO_ONE) ||
516516
!is_bare_repository())
517517
write_bitmaps = 0;
518+
} else if (write_bitmaps &&
519+
git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&
520+
git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {
521+
write_bitmaps = 0;
518522
}
519523
if (pack_kept_objects < 0)
520524
pack_kept_objects = write_bitmaps > 0;
@@ -725,8 +729,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
725729
update_server_info(0);
726730
remove_temporary_files();
727731

728-
if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))
729-
write_midx_file(get_object_directory(), NULL, 0);
732+
if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {
733+
unsigned flags = 0;
734+
if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))
735+
flags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;
736+
write_midx_file(get_object_directory(), NULL, flags);
737+
}
730738

731739
string_list_clear(&names, 0);
732740
string_list_clear(&rollback, 0);

‎ci/run-build-and-tests.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ linux-gcc)
2828
export GIT_TEST_COMMIT_GRAPH=1
2929
export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1
3030
export GIT_TEST_MULTI_PACK_INDEX=1
31+
export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1
3132
export GIT_TEST_ADD_I_USE_BUILTIN=1
3233
export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master
3334
export GIT_TEST_WRITE_REV_INDEX=1

‎git.c‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -561,7 +561,7 @@ static struct cmd_struct commands[] = {
561561
{ "merge-tree", cmd_merge_tree, RUN_SETUP | NO_PARSEOPT },
562562
{ "mktag", cmd_mktag, RUN_SETUP | NO_PARSEOPT },
563563
{ "mktree", cmd_mktree, RUN_SETUP },
564-
{ "multi-pack-index", cmd_multi_pack_index, RUN_SETUP_GENTLY },
564+
{ "multi-pack-index", cmd_multi_pack_index, RUN_SETUP },
565565
{ "mv", cmd_mv, RUN_SETUP | NEED_WORK_TREE },
566566
{ "name-rev", cmd_name_rev, RUN_SETUP },
567567
{ "notes", cmd_notes, RUN_SETUP },

0 commit comments

Comments
 (0)