Skip to content

Commit f3d618d

Browse files
gitstergitster
committed
Merge branch 'jh/fsck-promisors'
In preparation for implementing narrow/partial clone, the machinery for checking object connectivity used by gc and fsck has been taught that a missing object is OK when it is referenced by a packfile specially marked as coming from trusted repository that promises to make them available on-demand and lazily. * jh/fsck-promisors: gc: do not repack promisor packfiles rev-list: support termination at promisor objects sha1_file: support lazily fetching missing objects introduce fetch-object: fetch one promisor object index-pack: refactor writing of .keep files fsck: support promisor objects as CLI argument fsck: support referenced promisor objects fsck: support refs pointing to promisor objects fsck: introduce partialclone extension extension.partialclone: introduce partial clone extension
2 parents ed1b87e + 0c16cd4 commit f3d618d

32 files changed

+896
-96
lines changed

‎Documentation/git-pack-objects.txt‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,17 @@ a missing object is encountered. This is the default action.
255255
The form '--missing=allow-any' will allow object traversal to continue
256256
if a missing object is encountered. Missing objects will silently be
257257
omitted from the results.
258+
+
259+
The form '--missing=allow-promisor' is like 'allow-any', but will only
260+
allow object traversal to continue for EXPECTED promisor missing objects.
261+
Unexpected missing object will raise an error.
262+
263+
--exclude-promisor-objects::
264+
Omit objects that are known to be in the promisor remote. (This
265+
option has the purpose of operating only on locally created objects,
266+
so that when we repack, we still maintain a distinction between
267+
locally created objects [without .promisor] and objects from the
268+
promisor remote [with .promisor].) This is used with partial clone.
258269

259270
SEE ALSO
260271
--------

‎Documentation/gitremote-helpers.txt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -466,6 +466,13 @@ set by Git if the remote helper has the 'option' capability.
466466
Transmit <string> as a push option. As the push option
467467
must not contain LF or NUL characters, the string is not encoded.
468468

469+
'option from-promisor' {'true'|'false'}::
470+
Indicate that these objects are being fetched from a promisor.
471+
472+
'option no-dependents' {'true'|'false'}::
473+
Indicate that only the objects wanted need to be fetched, not
474+
their dependents.
475+
469476
SEE ALSO
470477
--------
471478
linkgit:git-remote[1]

‎Documentation/rev-list-options.txt‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -750,10 +750,21 @@ The form '--missing=allow-any' will allow object traversal to continue
750750
if a missing object is encountered. Missing objects will silently be
751751
omitted from the results.
752752
+
753+
The form '--missing=allow-promisor' is like 'allow-any', but will only
754+
allow object traversal to continue for EXPECTED promisor missing objects.
755+
Unexpected missing objects will raise an error.
756+
+
753757
The form '--missing=print' is like 'allow-any', but will also print a
754758
list of the missing objects. Object IDs are prefixed with a ``?'' character.
755759
endif::git-rev-list[]
756760

761+
--exclude-promisor-objects::
762+
(For internal use only.) Prefilter object traversal at
763+
promisor boundary. This is used with partial clone. This is
764+
stronger than `--missing=allow-promisor` because it limits the
765+
traversal, rather than just silencing errors about missing
766+
objects.
767+
757768
--no-walk[=(sorted|unsorted)]::
758769
Only show the given commits, but do not traverse their ancestors.
759770
This has no effect if a range is specified. If the argument

‎Documentation/technical/repository-version.txt‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,3 +86,15 @@ for testing format-1 compatibility.
8686
When the config key `extensions.preciousObjects` is set to `true`,
8787
objects in the repository MUST NOT be deleted (e.g., by `git-prune` or
8888
`git repack -d`).
89+
90+
`partialclone`
91+
~~~~~~~~~~~~~~
92+
93+
When the config key `extensions.partialclone` is set, it indicates
94+
that the repo was created with a partial clone (or later performed
95+
a partial fetch) and that the remote may have omitted sending
96+
certain unwanted objects. Such a remote is called a "promisor remote"
97+
and it promises that all such omitted objects can be fetched from it
98+
in the future.
99+
100+
The value of this key is the name of the promisor remote.

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -802,6 +802,7 @@ LIB_OBJS += ewah/ewah_bitmap.o
802802
LIB_OBJS += ewah/ewah_io.o
803803
LIB_OBJS += ewah/ewah_rlw.o
804804
LIB_OBJS += exec_cmd.o
805+
LIB_OBJS += fetch-object.o
805806
LIB_OBJS += fetch-pack.o
806807
LIB_OBJS += fsck.o
807808
LIB_OBJS += fsmonitor.o

‎builtin/cat-file.c‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -475,6 +475,8 @@ static int batch_objects(struct batch_options *opt)
475475

476476
for_each_loose_object(batch_loose_object, &sa, 0);
477477
for_each_packed_object(batch_packed_object, &sa, 0);
478+
if (repository_format_partial_clone)
479+
warning("This repository has extensions.partialClone set. Some objects may not be loaded.");
478480

479481
cb.opt = opt;
480482
cb.expand = &data;

‎builtin/fetch-pack.c‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
5353
struct oid_array shallow = OID_ARRAY_INIT;
5454
struct string_list deepen_not = STRING_LIST_INIT_DUP;
5555

56+
fetch_if_missing = 0;
57+
5658
packet_trace_identity("fetch-pack");
5759

5860
memset(&args, 0, sizeof(args));
@@ -143,6 +145,14 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
143145
args.update_shallow = 1;
144146
continue;
145147
}
148+
if (!strcmp("--from-promisor", arg)) {
149+
args.from_promisor = 1;
150+
continue;
151+
}
152+
if (!strcmp("--no-dependents", arg)) {
153+
args.no_dependents = 1;
154+
continue;
155+
}
146156
usage(fetch_pack_usage);
147157
}
148158
if (deepen_not.nr)

‎builtin/fsck.c‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,15 @@ static int mark_object(struct object *obj, int type, void *data, struct fsck_opt
149149
if (obj->flags & REACHABLE)
150150
return 0;
151151
obj->flags |= REACHABLE;
152+
153+
if (is_promisor_object(&obj->oid))
154+
/*
155+
* Further recursion does not need to be performed on this
156+
* object since it is a promisor object (so it does not need to
157+
* be added to "pending").
158+
*/
159+
return 0;
160+
152161
if (!(obj->flags & HAS_OBJ)) {
153162
if (parent && !has_object_file(&obj->oid)) {
154163
printf("broken link from %7s %s\n",
@@ -208,6 +217,8 @@ static void check_reachable_object(struct object *obj)
208217
* do a full fsck
209218
*/
210219
if (!(obj->flags & HAS_OBJ)) {
220+
if (is_promisor_object(&obj->oid))
221+
return;
211222
if (has_sha1_pack(obj->oid.hash))
212223
return; /* it is in pack - forget about it */
213224
printf("missing %s %s\n", printable_type(obj),
@@ -398,7 +409,7 @@ static void fsck_handle_reflog_oid(const char *refname, struct object_id *oid,
398409
xstrfmt("%s@{%"PRItime"}", refname, timestamp));
399410
obj->flags |= USED;
400411
mark_object_reachable(obj);
401-
} else {
412+
} else if (!is_promisor_object(oid)) {
402413
error("%s: invalid reflog entry %s", refname, oid_to_hex(oid));
403414
errors_found |= ERROR_REACHABLE;
404415
}
@@ -434,6 +445,14 @@ static int fsck_handle_ref(const char *refname, const struct object_id *oid,
434445

435446
obj = parse_object(oid);
436447
if (!obj) {
448+
if (is_promisor_object(oid)) {
449+
/*
450+
* Increment default_refs anyway, because this is a
451+
* valid ref.
452+
*/
453+
default_refs++;
454+
return 0;
455+
}
437456
error("%s: invalid sha1 pointer %s", refname, oid_to_hex(oid));
438457
errors_found |= ERROR_REACHABLE;
439458
/* We'll continue with the rest despite the error.. */
@@ -659,6 +678,9 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
659678
int i;
660679
struct alternate_object_database *alt;
661680

681+
/* fsck knows how to handle missing promisor objects */
682+
fetch_if_missing = 0;
683+
662684
errors_found = 0;
663685
check_replace_refs = 0;
664686

@@ -731,6 +753,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
731753
struct object *obj = lookup_object(oid.hash);
732754

733755
if (!obj || !(obj->flags & HAS_OBJ)) {
756+
if (is_promisor_object(&oid))
757+
continue;
734758
error("%s: object missing", oid_to_hex(&oid));
735759
errors_found |= ERROR_OBJECT;
736760
continue;

‎builtin/gc.c‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -458,6 +458,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
458458
argv_array_push(&prune, prune_expire);
459459
if (quiet)
460460
argv_array_push(&prune, "--no-progress");
461+
if (repository_format_partial_clone)
462+
argv_array_push(&prune,
463+
"--exclude-promisor-objects");
461464
if (run_command_v_opt(prune.argv, RUN_GIT_CMD))
462465
return error(FAILED_RUN, prune.argv[0]);
463466
}

‎builtin/index-pack.c‎

Lines changed: 66 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1389,15 +1389,60 @@ static void fix_unresolved_deltas(struct sha1file *f)
13891389
free(sorted_by_pos);
13901390
}
13911391

1392+
static const char *derive_filename(const char *pack_name, const char *suffix,
1393+
struct strbuf *buf)
1394+
{
1395+
size_t len;
1396+
if (!strip_suffix(pack_name, ".pack", &len))
1397+
die(_("packfile name '%s' does not end with '.pack'"),
1398+
pack_name);
1399+
strbuf_add(buf, pack_name, len);
1400+
strbuf_addch(buf, '.');
1401+
strbuf_addstr(buf, suffix);
1402+
return buf->buf;
1403+
}
1404+
1405+
static void write_special_file(const char *suffix, const char *msg,
1406+
const char *pack_name, const unsigned char *sha1,
1407+
const char **report)
1408+
{
1409+
struct strbuf name_buf = STRBUF_INIT;
1410+
const char *filename;
1411+
int fd;
1412+
int msg_len = strlen(msg);
1413+
1414+
if (pack_name)
1415+
filename = derive_filename(pack_name, suffix, &name_buf);
1416+
else
1417+
filename = odb_pack_name(&name_buf, sha1, suffix);
1418+
1419+
fd = odb_pack_keep(filename);
1420+
if (fd < 0) {
1421+
if (errno != EEXIST)
1422+
die_errno(_("cannot write %s file '%s'"),
1423+
suffix, filename);
1424+
} else {
1425+
if (msg_len > 0) {
1426+
write_or_die(fd, msg, msg_len);
1427+
write_or_die(fd, "\n", 1);
1428+
}
1429+
if (close(fd) != 0)
1430+
die_errno(_("cannot close written %s file '%s'"),
1431+
suffix, filename);
1432+
if (report)
1433+
*report = suffix;
1434+
}
1435+
strbuf_release(&name_buf);
1436+
}
1437+
13921438
static void final(const char *final_pack_name, const char *curr_pack_name,
13931439
const char *final_index_name, const char *curr_index_name,
1394-
const char *keep_name, const char *keep_msg,
1440+
const char *keep_msg, const char *promisor_msg,
13951441
unsigned char *sha1)
13961442
{
13971443
const char *report = "pack";
13981444
struct strbuf pack_name = STRBUF_INIT;
13991445
struct strbuf index_name = STRBUF_INIT;
1400-
struct strbuf keep_name_buf = STRBUF_INIT;
14011446
int err;
14021447

14031448
if (!from_stdin) {
@@ -1409,28 +1454,12 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
14091454
die_errno(_("error while closing pack file"));
14101455
}
14111456

1412-
if (keep_msg) {
1413-
int keep_fd, keep_msg_len = strlen(keep_msg);
1414-
1415-
if (!keep_name)
1416-
keep_name = odb_pack_name(&keep_name_buf, sha1, "keep");
1417-
1418-
keep_fd = odb_pack_keep(keep_name);
1419-
if (keep_fd < 0) {
1420-
if (errno != EEXIST)
1421-
die_errno(_("cannot write keep file '%s'"),
1422-
keep_name);
1423-
} else {
1424-
if (keep_msg_len > 0) {
1425-
write_or_die(keep_fd, keep_msg, keep_msg_len);
1426-
write_or_die(keep_fd, "\n", 1);
1427-
}
1428-
if (close(keep_fd) != 0)
1429-
die_errno(_("cannot close written keep file '%s'"),
1430-
keep_name);
1431-
report = "keep";
1432-
}
1433-
}
1457+
if (keep_msg)
1458+
write_special_file("keep", keep_msg, final_pack_name, sha1,
1459+
&report);
1460+
if (promisor_msg)
1461+
write_special_file("promisor", promisor_msg, final_pack_name,
1462+
sha1, NULL);
14341463

14351464
if (final_pack_name != curr_pack_name) {
14361465
if (!final_pack_name)
@@ -1472,7 +1501,6 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
14721501

14731502
strbuf_release(&index_name);
14741503
strbuf_release(&pack_name);
1475-
strbuf_release(&keep_name_buf);
14761504
}
14771505

14781506
static int git_index_pack_config(const char *k, const char *v, void *cb)
@@ -1615,32 +1643,26 @@ static void show_pack_info(int stat_only)
16151643
}
16161644
}
16171645

1618-
static const char *derive_filename(const char *pack_name, const char *suffix,
1619-
struct strbuf *buf)
1620-
{
1621-
size_t len;
1622-
if (!strip_suffix(pack_name, ".pack", &len))
1623-
die(_("packfile name '%s' does not end with '.pack'"),
1624-
pack_name);
1625-
strbuf_add(buf, pack_name, len);
1626-
strbuf_addstr(buf, suffix);
1627-
return buf->buf;
1628-
}
1629-
16301646
int cmd_index_pack(int argc, const char **argv, const char *prefix)
16311647
{
16321648
int i, fix_thin_pack = 0, verify = 0, stat_only = 0;
16331649
const char *curr_index;
16341650
const char *index_name = NULL, *pack_name = NULL;
1635-
const char *keep_name = NULL, *keep_msg = NULL;
1636-
struct strbuf index_name_buf = STRBUF_INIT,
1637-
keep_name_buf = STRBUF_INIT;
1651+
const char *keep_msg = NULL;
1652+
const char *promisor_msg = NULL;
1653+
struct strbuf index_name_buf = STRBUF_INIT;
16381654
struct pack_idx_entry **idx_objects;
16391655
struct pack_idx_option opts;
16401656
unsigned char pack_sha1[20];
16411657
unsigned foreign_nr = 1; /* zero is a "good" value, assume bad */
16421658
int report_end_of_input = 0;
16431659

1660+
/*
1661+
* index-pack never needs to fetch missing objects, since it only
1662+
* accesses the repo to do hash collision checks
1663+
*/
1664+
fetch_if_missing = 0;
1665+
16441666
if (argc == 2 && !strcmp(argv[1], "-h"))
16451667
usage(index_pack_usage);
16461668

@@ -1678,6 +1700,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
16781700
stat_only = 1;
16791701
} else if (skip_to_optional_arg(arg, "--keep", &keep_msg)) {
16801702
; /* nothing to do */
1703+
} else if (skip_to_optional_arg(arg, "--promisor", &promisor_msg)) {
1704+
; /* already parsed */
16811705
} else if (starts_with(arg, "--threads=")) {
16821706
char *end;
16831707
nr_threads = strtoul(arg+10, &end, 0);
@@ -1740,9 +1764,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
17401764
if (from_stdin && !startup_info->have_repository)
17411765
die(_("--stdin requires a git repository"));
17421766
if (!index_name && pack_name)
1743-
index_name = derive_filename(pack_name, ".idx", &index_name_buf);
1744-
if (keep_msg && !keep_name && pack_name)
1745-
keep_name = derive_filename(pack_name, ".keep", &keep_name_buf);
1767+
index_name = derive_filename(pack_name, "idx", &index_name_buf);
17461768

17471769
if (verify) {
17481770
if (!index_name)
@@ -1790,13 +1812,12 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
17901812
if (!verify)
17911813
final(pack_name, curr_pack,
17921814
index_name, curr_index,
1793-
keep_name, keep_msg,
1815+
keep_msg, promisor_msg,
17941816
pack_sha1);
17951817
else
17961818
close(input_fd);
17971819
free(objects);
17981820
strbuf_release(&index_name_buf);
1799-
strbuf_release(&keep_name_buf);
18001821
if (pack_name == NULL)
18011822
free((void *) curr_pack);
18021823
if (index_name == NULL)

0 commit comments

Comments
 (0)