Skip to content

Commit 7391ecd

Browse files
gitstergitster
committed
Merge branch 'ds/partial-bundles'
Bundle file format gets extended to allow a partial bundle, filtered by similar criteria you would give when making a partial/lazy clone. * ds/partial-bundles: clone: fail gracefully when cloning filtered bundle bundle: unbundle promisor packs bundle: create filtered bundles rev-list: move --filter parsing into revision.c bundle: parse filter capability list-objects: handle NULL function pointers MyFirstObjectWalk: update recommended usage list-objects: consolidate traverse_commit_list[_filtered] pack-bitmap: drop filter in prepare_bitmap_walk() pack-objects: use rev.filter when possible revision: put object filter into struct rev_info list-objects-filter-options: create copy helper index-pack: document and test the --promisor option
2 parents b6763af + 86fdd94 commit 7391ecd

20 files changed

+317
-133
lines changed

‎Documentation/MyFirstObjectWalk.txt‎

Lines changed: 16 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -522,24 +522,25 @@ function shows that the all-object walk is being performed by
522522
`traverse_commit_list()` or `traverse_commit_list_filtered()`. Those two
523523
functions reside in `list-objects.c`; examining the source shows that, despite
524524
the name, these functions traverse all kinds of objects. Let's have a look at
525-
the arguments to `traverse_commit_list_filtered()`, which are a superset of the
526-
arguments to the unfiltered version.
525+
the arguments to `traverse_commit_list()`.
527526

528-
- `struct list_objects_filter_options *filter_options`: This is a struct which
529-
stores a filter-spec as outlined in `Documentation/rev-list-options.txt`.
530-
- `struct rev_info *revs`: This is the `rev_info` used for the walk.
527+
- `struct rev_info *revs`: This is the `rev_info` used for the walk. If
528+
its `filter` member is not `NULL`, then `filter` contains information for
529+
how to filter the object list.
531530
- `show_commit_fn show_commit`: A callback which will be used to handle each
532531
individual commit object.
533532
- `show_object_fn show_object`: A callback which will be used to handle each
534533
non-commit object (so each blob, tree, or tag).
535534
- `void *show_data`: A context buffer which is passed in turn to `show_commit`
536535
and `show_object`.
536+
537+
In addition, `traverse_commit_list_filtered()` has an additional paramter:
538+
537539
- `struct oidset *omitted`: A linked-list of object IDs which the provided
538540
filter caused to be omitted.
539541

540-
It looks like this `traverse_commit_list_filtered()` uses callbacks we provide
541-
instead of needing us to call it repeatedly ourselves. Cool! Let's add the
542-
callbacks first.
542+
It looks like these methods use callbacks we provide instead of needing us
543+
to call it repeatedly ourselves. Cool! Let's add the callbacks first.
543544

544545
For the sake of this tutorial, we'll simply keep track of how many of each kind
545546
of object we find. At file scope in `builtin/walken.c` add the following
@@ -712,40 +713,27 @@ help understand. In our case, that means we omit trees and blobs not directly
712713
referenced by `HEAD` or `HEAD`'s history, because we begin the walk with only
713714
`HEAD` in the `pending` list.)
714715

715-
First, we'll need to `#include "list-objects-filter-options.h"` and set up the
716-
`struct list_objects_filter_options` at the top of the function.
717-
718-
----
719-
static void walken_object_walk(struct rev_info *rev)
720-
{
721-
struct list_objects_filter_options filter_options = { 0 };
722-
723-
...
724-
----
725-
726716
For now, we are not going to track the omitted objects, so we'll replace those
727717
parameters with `NULL`. For the sake of simplicity, we'll add a simple
728-
build-time branch to use our filter or not. Replace the line calling
718+
build-time branch to use our filter or not. Preface the line calling
729719
`traverse_commit_list()` with the following, which will remind us which kind of
730720
walk we've just performed:
731721

732722
----
733723
if (0) {
734724
/* Unfiltered: */
735725
trace_printf(_("Unfiltered object walk.\n"));
736-
traverse_commit_list(rev, walken_show_commit,
737-
walken_show_object, NULL);
738726
} else {
739727
trace_printf(
740728
_("Filtered object walk with filterspec 'tree:1'.\n"));
741-
parse_list_objects_filter(&filter_options, "tree:1");
742-
743-
traverse_commit_list_filtered(&filter_options, rev,
744-
walken_show_commit, walken_show_object, NULL, NULL);
729+
CALLOC_ARRAY(rev->filter, 1);
730+
parse_list_objects_filter(rev->filter, "tree:1");
745731
}
732+
traverse_commit_list(rev, walken_show_commit,
733+
walken_show_object, NULL);
746734
----
747735

748-
`struct list_objects_filter_options` is usually built directly from a command
736+
The `rev->filter` member is usually built directly from a command
749737
line argument, so the module provides an easy way to build one from a string.
750738
Even though we aren't taking user input right now, we can still build one with
751739
a hardcoded string using `parse_list_objects_filter()`.
@@ -784,7 +772,7 @@ object:
784772
----
785773
...
786774

787-
traverse_commit_list_filtered(&filter_options, rev,
775+
traverse_commit_list_filtered(rev,
788776
walken_show_commit, walken_show_object, NULL, &omitted);
789777

790778
...

‎Documentation/git-bundle.txt‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,11 @@ verify <file>::
7575
cleanly to the current repository. This includes checks on the
7676
bundle format itself as well as checking that the prerequisite
7777
commits exist and are fully linked in the current repository.
78-
'git bundle' prints a list of missing commits, if any, and exits
79-
with a non-zero status.
78+
Information about additional capabilities, such as "object filter",
79+
is printed. See "Capabilities" in link:technical/bundle-format.html
80+
for more information. Finally, 'git bundle' prints a list of
81+
missing commits, if any. The exit code is zero for success, but
82+
will be nonzero if the bundle file is invalid.
8083

8184
list-heads <file>::
8285
Lists the references defined in the bundle. If followed by a

‎Documentation/git-index-pack.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,14 @@ This option cannot be used with --stdin.
122122
+
123123
include::object-format-disclaimer.txt[]
124124

125+
--promisor[=<message>]::
126+
Before committing the pack-index, create a .promisor file for this
127+
pack. Particularly helpful when writing a promisor pack with --fix-thin
128+
since the name of the pack is not final until the pack has been fully
129+
written. If a `<message>` is provided, then that content will be
130+
written to the .promisor file for future reference. See
131+
link:technical/partial-clone.html[partial clone] for more information.
132+
125133
NOTES
126134
-----
127135

‎Documentation/technical/bundle-format.txt‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,11 @@ and the Git bundle v2 format cannot represent a shallow clone repository.
7171
== Capabilities
7272

7373
Because there is no opportunity for negotiation, unknown capabilities cause 'git
74-
bundle' to abort. The only known capability is `object-format`, which specifies
75-
the hash algorithm in use, and can take the same values as the
76-
`extensions.objectFormat` configuration value.
74+
bundle' to abort.
75+
76+
* `object-format` specifies the hash algorithm in use, and can take the same
77+
values as the `extensions.objectFormat` configuration value.
78+
79+
* `filter` specifies an object filter as in the `--filter` option in
80+
linkgit:git-rev-list[1]. The resulting pack-file must be marked as a
81+
`.promisor` pack-file after it is unbundled.

‎builtin/clone.c‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
#include "packfile.h"
3434
#include "list-objects-filter-options.h"
3535
#include "hook.h"
36+
#include "bundle.h"
3637

3738
/*
3839
* Overall FIXMEs:
@@ -1172,6 +1173,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
11721173
warning(_("--local is ignored"));
11731174
transport->cloning = 1;
11741175

1176+
if (is_bundle) {
1177+
struct bundle_header header = BUNDLE_HEADER_INIT;
1178+
int fd = read_bundle_header(path, &header);
1179+
int has_filter = header.filter.choice != LOFC_DISABLED;
1180+
1181+
if (fd > 0)
1182+
close(fd);
1183+
bundle_header_release(&header);
1184+
if (has_filter)
1185+
die(_("cannot clone from filtered bundle"));
1186+
}
1187+
11751188
transport_set_option(transport, TRANS_OPT_KEEP, "yes");
11761189

11771190
if (reject_shallow)

‎builtin/pack-objects.c‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)
36513651

36523652
static int get_object_list_from_bitmap(struct rev_info *revs)
36533653
{
3654-
if (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))
3654+
if (!(bitmap_git = prepare_bitmap_walk(revs, 0)))
36553655
return -1;
36563656

36573657
if (pack_options_allow_reuse() &&
@@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av)
37273727
repo_init_revisions(the_repository, &revs, NULL);
37283728
save_commit_buffer = 0;
37293729
setup_revisions(ac, av, &revs, &s_r_opt);
3730+
list_objects_filter_copy(&revs.filter, &filter_options);
37303731

37313732
/* make sure shallows are read */
37323733
is_repository_shallow(the_repository);
@@ -3777,9 +3778,9 @@ static void get_object_list(int ac, const char **av)
37773778

37783779
if (!fn_show_object)
37793780
fn_show_object = show_object;
3780-
traverse_commit_list_filtered(&filter_options, &revs,
3781-
show_commit, fn_show_object, NULL,
3782-
NULL);
3781+
traverse_commit_list(&revs,
3782+
show_commit, fn_show_object,
3783+
NULL);
37833784

37843785
if (unpack_unreachable_expiration) {
37853786
revs.ignore_missing_links = 1;

‎builtin/rev-list.c‎

Lines changed: 7 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,6 @@ static const char rev_list_usage[] =
6262
static struct progress *progress;
6363
static unsigned progress_counter;
6464

65-
static struct list_objects_filter_options filter_options;
6665
static struct oidset omitted_objects;
6766
static int arg_print_omitted; /* print objects omitted by filter */
6867

@@ -400,7 +399,6 @@ static inline int parse_missing_action_value(const char *value)
400399
}
401400

402401
static int try_bitmap_count(struct rev_info *revs,
403-
struct list_objects_filter_options *filter,
404402
int filter_provided_objects)
405403
{
406404
uint32_t commit_count = 0,
@@ -436,7 +434,7 @@ static int try_bitmap_count(struct rev_info *revs,
436434
*/
437435
max_count = revs->max_count;
438436

439-
bitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);
437+
bitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);
440438
if (!bitmap_git)
441439
return -1;
442440

@@ -453,7 +451,6 @@ static int try_bitmap_count(struct rev_info *revs,
453451
}
454452

455453
static int try_bitmap_traversal(struct rev_info *revs,
456-
struct list_objects_filter_options *filter,
457454
int filter_provided_objects)
458455
{
459456
struct bitmap_index *bitmap_git;
@@ -465,7 +462,7 @@ static int try_bitmap_traversal(struct rev_info *revs,
465462
if (revs->max_count >= 0)
466463
return -1;
467464

468-
bitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);
465+
bitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);
469466
if (!bitmap_git)
470467
return -1;
471468

@@ -475,15 +472,14 @@ static int try_bitmap_traversal(struct rev_info *revs,
475472
}
476473

477474
static int try_bitmap_disk_usage(struct rev_info *revs,
478-
struct list_objects_filter_options *filter,
479475
int filter_provided_objects)
480476
{
481477
struct bitmap_index *bitmap_git;
482478

483479
if (!show_disk_usage)
484480
return -1;
485481

486-
bitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);
482+
bitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);
487483
if (!bitmap_git)
488484
return -1;
489485

@@ -595,17 +591,6 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)
595591
show_progress = arg;
596592
continue;
597593
}
598-
599-
if (skip_prefix(arg, ("--" CL_ARG__FILTER "="), &arg)) {
600-
parse_list_objects_filter(&filter_options, arg);
601-
if (filter_options.choice && !revs.blob_objects)
602-
die(_("object filtering requires --objects"));
603-
continue;
604-
}
605-
if (!strcmp(arg, ("--no-" CL_ARG__FILTER))) {
606-
list_objects_filter_set_no_filter(&filter_options);
607-
continue;
608-
}
609594
if (!strcmp(arg, "--filter-provided-objects")) {
610595
filter_provided_objects = 1;
611596
continue;
@@ -688,11 +673,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)
688673
progress = start_delayed_progress(show_progress, 0);
689674

690675
if (use_bitmap_index) {
691-
if (!try_bitmap_count(&revs, &filter_options, filter_provided_objects))
676+
if (!try_bitmap_count(&revs, filter_provided_objects))
692677
return 0;
693-
if (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_objects))
678+
if (!try_bitmap_disk_usage(&revs, filter_provided_objects))
694679
return 0;
695-
if (!try_bitmap_traversal(&revs, &filter_options, filter_provided_objects))
680+
if (!try_bitmap_traversal(&revs, filter_provided_objects))
696681
return 0;
697682
}
698683

@@ -733,7 +718,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)
733718
oidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);
734719

735720
traverse_commit_list_filtered(
736-
&filter_options, &revs, show_commit, show_object, &info,
721+
&revs, show_commit, show_object, &info,
737722
(arg_print_omitted ? &omitted_objects : NULL));
738723

739724
if (arg_print_omitted) {

0 commit comments

Comments
 (0)