Skip to content

Commit a460ea4

Browse files
gitstergitster
committed
Merge branch 'nd/shallow-deepen'
The existing "git fetch --depth=<n>" option was hard to use correctly when making the history of an existing shallow clone deeper. A new option, "--deepen=<n>", has been added to make this easier to use. "git clone" also learned "--shallow-since=<date>" and "--shallow-exclude=<tag>" options to make it easier to specify "I am interested only in the recent N months worth of history" and "Give me only the history since that version". * nd/shallow-deepen: (27 commits) fetch, upload-pack: --deepen=N extends shallow boundary by N commits upload-pack: add get_reachable_list() upload-pack: split check_unreachable() in two, prep for get_reachable_list() t5500, t5539: tests for shallow depth excluding a ref clone: define shallow clone boundary with --shallow-exclude fetch: define shallow boundary with --shallow-exclude upload-pack: support define shallow boundary by excluding revisions refs: add expand_ref() t5500, t5539: tests for shallow depth since a specific date clone: define shallow clone boundary based on time with --shallow-since fetch: define shallow boundary with --shallow-since upload-pack: add deepen-since to cut shallow repos based on time shallow.c: implement a generic shallow boundary finder based on rev-list fetch-pack: use a separate flag for fetch in deepening mode fetch-pack.c: mark strings for translating fetch-pack: use a common function for verbose printing fetch-pack: use skip_prefix() instead of starts_with() upload-pack: move rev-list code out of check_non_tip() upload-pack: make check_non_tip() clean things up on error upload-pack: tighten number parsing at "deepen" lines ...
2 parents a229a30 + cccf74e commit a460ea4

23 files changed

+890
-224
lines changed

‎Documentation/fetch-options.txt‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,20 @@
1414
linkgit:git-clone[1]), deepen or shorten the history to the specified
1515
number of commits. Tags for the deepened commits are not fetched.
1616

17+
--deepen=<depth>::
18+
Similar to --depth, except it specifies the number of commits
19+
from the current shallow boundary instead of from the tip of
20+
each remote branch history.
21+
22+
--shallow-since=<date>::
23+
Deepen or shorten the history of a shallow repository to
24+
include all reachable commits after <date>.
25+
26+
--shallow-exclude=<revision>::
27+
Deepen or shorten the history of a shallow repository to
28+
exclude commits reachable from a specified remote branch or tag.
29+
This option can be specified multiple times.
30+
1731
--unshallow::
1832
If the source repository is complete, convert a shallow
1933
repository to a complete one, removing all the limitations

‎Documentation/git-clone.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,14 @@ objects from the source repository into a pack in the cloned repository.
197197
tips of all branches. If you want to clone submodules shallowly,
198198
also pass `--shallow-submodules`.
199199

200+
--shallow-since=<date>::
201+
Create a shallow clone with a history after the specified time.
202+
203+
--shallow-exclude=<revision>::
204+
Create a shallow clone with a history, excluding commits
205+
reachable from a specified remote branch or tag. This option
206+
can be specified multiple times.
207+
200208
--[no-]single-branch::
201209
Clone only the history leading to the tip of a single branch,
202210
either specified by the `--branch` option or the primary

‎Documentation/git-fetch-pack.txt‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,20 @@ be in a separate packet, and the list must end with a flush packet.
8787
'git-upload-pack' treats the special depth 2147483647 as
8888
infinite even if there is an ancestor-chain that long.
8989

90+
--shallow-since=<date>::
91+
Deepen or shorten the history of a shallow'repository to
92+
include all reachable commits after <date>.
93+
94+
--shallow-exclude=<revision>::
95+
Deepen or shorten the history of a shallow repository to
96+
exclude commits reachable from a specified remote branch or tag.
97+
This option can be specified multiple times.
98+
99+
--deepen-relative::
100+
Argument --depth specifies the number of commits from the
101+
current shallow boundary instead of from the tip of each
102+
remote branch history.
103+
90104
--no-progress::
91105
Do not show the progress.
92106

‎Documentation/gitremote-helpers.txt‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -415,6 +415,17 @@ set by Git if the remote helper has the 'option' capability.
415415
'option depth' <depth>::
416416
Deepens the history of a shallow repository.
417417

418+
'option deepen-since <timestamp>::
419+
Deepens the history of a shallow repository based on time.
420+
421+
'option deepen-not <ref>::
422+
Deepens the history of a shallow repository excluding ref.
423+
Multiple options add up.
424+
425+
'option deepen-relative {'true'|'false'}::
426+
Deepens the history of a shallow repository relative to
427+
current boundary. Only valid when used with "option depth".
428+
418429
'option followtags' {'true'|'false'}::
419430
If enabled the helper should automatically fetch annotated
420431
tag objects if the object the tag points at was transferred

‎Documentation/technical/pack-protocol.txt‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,7 +219,9 @@ out of what the server said it could do with the first 'want' line.
219219

220220
shallow-line = PKT-LINE("shallow" SP obj-id)
221221

222-
depth-request = PKT-LINE("deepen" SP depth)
222+
depth-request = PKT-LINE("deepen" SP depth) /
223+
PKT-LINE("deepen-since" SP timestamp) /
224+
PKT-LINE("deepen-not" SP ref)
223225

224226
first-want = PKT-LINE("want" SP obj-id SP capability-list)
225227
additional-want = PKT-LINE("want" SP obj-id)

‎Documentation/technical/protocol-capabilities.txt‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,31 @@ This capability adds "deepen", "shallow" and "unshallow" commands to
179179
the fetch-pack/upload-pack protocol so clients can request shallow
180180
clones.
181181

182+
deepen-since
183+
------------
184+
185+
This capability adds "deepen-since" command to fetch-pack/upload-pack
186+
protocol so the client can request shallow clones that are cut at a
187+
specific time, instead of depth. Internally it's equivalent of doing
188+
"rev-list --max-age=<timestamp>" on the server side. "deepen-since"
189+
cannot be used with "deepen".
190+
191+
deepen-not
192+
----------
193+
194+
This capability adds "deepen-not" command to fetch-pack/upload-pack
195+
protocol so the client can request shallow clones that are cut at a
196+
specific revision, instead of depth. Internally it's equivalent of
197+
doing "rev-list --not <rev>" on the server side. "deepen-not"
198+
cannot be used with "deepen", but can be used with "deepen-since".
199+
200+
deepen-relative
201+
---------------
202+
203+
If this capability is requested by the client, the semantics of
204+
"deepen" command is changed. The "depth" argument is the depth from
205+
the current shallow boundary, instead of the depth from remote refs.
206+
182207
no-progress
183208
-----------
184209

‎builtin/clone.c‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,11 @@ static const char * const builtin_clone_usage[] = {
4141
static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
4242
static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
4343
static int option_shallow_submodules;
44-
static char *option_template, *option_depth;
44+
static int deepen;
45+
static char *option_template, *option_depth, *option_since;
4546
static char *option_origin = NULL;
4647
static char *option_branch = NULL;
48+
static struct string_list option_not = STRING_LIST_INIT_NODUP;
4749
static const char *real_git_dir;
4850
static char *option_upload_pack = "git-upload-pack";
4951
static int option_verbosity;
@@ -94,6 +96,10 @@ static struct option builtin_clone_options[] = {
9496
N_("path to git-upload-pack on the remote")),
9597
OPT_STRING(0, "depth", &option_depth, N_("depth"),
9698
N_("create a shallow clone of that depth")),
99+
OPT_STRING(0, "shallow-since", &option_since, N_("time"),
100+
N_("create a shallow clone since a specific time")),
101+
OPT_STRING_LIST(0, "shallow-exclude", &option_not, N_("revision"),
102+
N_("deepen history of shallow clone by excluding rev")),
97103
OPT_BOOL(0, "single-branch", &option_single_branch,
98104
N_("clone only one branch, HEAD or --branch")),
99105
OPT_BOOL(0, "shallow-submodules", &option_shallow_submodules,
@@ -861,8 +867,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
861867
usage_msg_opt(_("You must specify a repository to clone."),
862868
builtin_clone_usage, builtin_clone_options);
863869

870+
if (option_depth || option_since || option_not.nr)
871+
deepen = 1;
864872
if (option_single_branch == -1)
865-
option_single_branch = option_depth ? 1 : 0;
873+
option_single_branch = deepen ? 1 : 0;
866874

867875
if (option_mirror)
868876
option_bare = 1;
@@ -1006,6 +1014,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
10061014
if (is_local) {
10071015
if (option_depth)
10081016
warning(_("--depth is ignored in local clones; use file:// instead."));
1017+
if (option_since)
1018+
warning(_("--shallow-since is ignored in local clones; use file:// instead."));
1019+
if (option_not.nr)
1020+
warning(_("--shallow-exclude is ignored in local clones; use file:// instead."));
10091021
if (!access(mkpath("%s/shallow", path), F_OK)) {
10101022
if (option_local > 0)
10111023
warning(_("source repository is shallow, ignoring --local"));
@@ -1024,14 +1036,20 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
10241036
if (option_depth)
10251037
transport_set_option(transport, TRANS_OPT_DEPTH,
10261038
option_depth);
1039+
if (option_since)
1040+
transport_set_option(transport, TRANS_OPT_DEEPEN_SINCE,
1041+
option_since);
1042+
if (option_not.nr)
1043+
transport_set_option(transport, TRANS_OPT_DEEPEN_NOT,
1044+
(const char *)&option_not);
10271045
if (option_single_branch)
10281046
transport_set_option(transport, TRANS_OPT_FOLLOWTAGS, "1");
10291047

10301048
if (option_upload_pack)
10311049
transport_set_option(transport, TRANS_OPT_UPLOADPACK,
10321050
option_upload_pack);
10331051

1034-
if (transport->smart_options && !option_depth)
1052+
if (transport->smart_options && !deepen)
10351053
transport->smart_options->check_self_contained_and_connected = 1;
10361054

10371055
refs = transport_get_remote_refs(transport);

‎builtin/fetch-pack.c‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
5151
struct child_process *conn;
5252
struct fetch_pack_args args;
5353
struct sha1_array shallow = SHA1_ARRAY_INIT;
54+
struct string_list deepen_not = STRING_LIST_INIT_DUP;
5455

5556
packet_trace_identity("fetch-pack");
5657

@@ -60,12 +61,12 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
6061
for (i = 1; i < argc && *argv[i] == '-'; i++) {
6162
const char *arg = argv[i];
6263

63-
if (starts_with(arg, "--upload-pack=")) {
64-
args.uploadpack = arg + 14;
64+
if (skip_prefix(arg, "--upload-pack=", &arg)) {
65+
args.uploadpack = arg;
6566
continue;
6667
}
67-
if (starts_with(arg, "--exec=")) {
68-
args.uploadpack = arg + 7;
68+
if (skip_prefix(arg, "--exec=", &arg)) {
69+
args.uploadpack = arg;
6970
continue;
7071
}
7172
if (!strcmp("--quiet", arg) || !strcmp("-q", arg)) {
@@ -101,8 +102,20 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
101102
args.verbose = 1;
102103
continue;
103104
}
104-
if (starts_with(arg, "--depth=")) {
105-
args.depth = strtol(arg + 8, NULL, 0);
105+
if (skip_prefix(arg, "--depth=", &arg)) {
106+
args.depth = strtol(arg, NULL, 0);
107+
continue;
108+
}
109+
if (skip_prefix(arg, "--shallow-since=", &arg)) {
110+
args.deepen_since = xstrdup(arg);
111+
continue;
112+
}
113+
if (skip_prefix(arg, "--shallow-exclude=", &arg)) {
114+
string_list_append(&deepen_not, arg);
115+
continue;
116+
}
117+
if (!strcmp(arg, "--deepen-relative")) {
118+
args.deepen_relative = 1;
106119
continue;
107120
}
108121
if (!strcmp("--no-progress", arg)) {
@@ -132,6 +145,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
132145
}
133146
usage(fetch_pack_usage);
134147
}
148+
if (deepen_not.nr)
149+
args.deepen_not = &deepen_not;
135150

136151
if (i < argc)
137152
dest = argv[i++];

‎builtin/fetch.c‎

Lines changed: 43 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -35,13 +35,15 @@ static int fetch_prune_config = -1; /* unspecified */
3535
static int prune = -1; /* unspecified */
3636
#define PRUNE_BY_DEFAULT 0 /* do we prune by default? */
3737

38-
static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity;
38+
static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity, deepen_relative;
3939
static int progress = -1, recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
40-
static int tags = TAGS_DEFAULT, unshallow, update_shallow;
40+
static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
4141
static int max_children = -1;
4242
static enum transport_family family;
4343
static const char *depth;
44+
static const char *deepen_since;
4445
static const char *upload_pack;
46+
static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
4547
static struct strbuf default_rla = STRBUF_INIT;
4648
static struct transport *gtransport;
4749
static struct transport *gsecondary;
@@ -117,6 +119,12 @@ static struct option builtin_fetch_options[] = {
117119
OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
118120
OPT_STRING(0, "depth", &depth, N_("depth"),
119121
N_("deepen history of shallow clone")),
122+
OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
123+
N_("deepen history of shallow repository based on time")),
124+
OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
125+
N_("deepen history of shallow clone by excluding rev")),
126+
OPT_INTEGER(0, "deepen", &deepen_relative,
127+
N_("deepen history of shallow clone")),
120128
{ OPTION_SET_INT, 0, "unshallow", &unshallow, NULL,
121129
N_("convert to a complete repository"),
122130
PARSE_OPT_NONEG | PARSE_OPT_NOARG, NULL, 1 },
@@ -875,7 +883,7 @@ static int quickfetch(struct ref *ref_map)
875883
* really need to perform. Claiming failure now will ensure
876884
* we perform the network exchange to deepen our history.
877885
*/
878-
if (depth)
886+
if (deepen)
879887
return -1;
880888
opt.quiet = 1;
881889
return check_connected(iterate_ref_map, &rm, &opt);
@@ -983,7 +991,7 @@ static void set_option(struct transport *transport, const char *name, const char
983991
name, transport->url);
984992
}
985993

986-
static struct transport *prepare_transport(struct remote *remote)
994+
static struct transport *prepare_transport(struct remote *remote, int deepen)
987995
{
988996
struct transport *transport;
989997
transport = transport_get(remote, NULL);
@@ -995,20 +1003,39 @@ static struct transport *prepare_transport(struct remote *remote)
9951003
set_option(transport, TRANS_OPT_KEEP, "yes");
9961004
if (depth)
9971005
set_option(transport, TRANS_OPT_DEPTH, depth);
1006+
if (deepen && deepen_since)
1007+
set_option(transport, TRANS_OPT_DEEPEN_SINCE, deepen_since);
1008+
if (deepen && deepen_not.nr)
1009+
set_option(transport, TRANS_OPT_DEEPEN_NOT,
1010+
(const char *)&deepen_not);
1011+
if (deepen_relative)
1012+
set_option(transport, TRANS_OPT_DEEPEN_RELATIVE, "yes");
9981013
if (update_shallow)
9991014
set_option(transport, TRANS_OPT_UPDATE_SHALLOW, "yes");
10001015
return transport;
10011016
}
10021017

10031018
static void backfill_tags(struct transport *transport, struct ref *ref_map)
10041019
{
1005-
if (transport->cannot_reuse) {
1006-
gsecondary = prepare_transport(transport->remote);
1020+
int cannot_reuse;
1021+
1022+
/*
1023+
* Once we have set TRANS_OPT_DEEPEN_SINCE, we can't unset it
1024+
* when remote helper is used (setting it to an empty string
1025+
* is not unsetting). We could extend the remote helper
1026+
* protocol for that, but for now, just force a new connection
1027+
* without deepen-since. Similar story for deepen-not.
1028+
*/
1029+
cannot_reuse = transport->cannot_reuse ||
1030+
deepen_since || deepen_not.nr;
1031+
if (cannot_reuse) {
1032+
gsecondary = prepare_transport(transport->remote, 0);
10071033
transport = gsecondary;
10081034
}
10091035

10101036
transport_set_option(transport, TRANS_OPT_FOLLOWTAGS, NULL);
10111037
transport_set_option(transport, TRANS_OPT_DEPTH, "0");
1038+
transport_set_option(transport, TRANS_OPT_DEEPEN_RELATIVE, NULL);
10121039
fetch_refs(transport, ref_map);
10131040

10141041
if (gsecondary) {
@@ -1219,7 +1246,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv)
12191246
die(_("No remote repository specified. Please, specify either a URL or a\n"
12201247
"remote name from which new revisions should be fetched."));
12211248

1222-
gtransport = prepare_transport(remote);
1249+
gtransport = prepare_transport(remote, 1);
12231250

12241251
if (prune < 0) {
12251252
/* no command line request */
@@ -1279,6 +1306,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
12791306
argc = parse_options(argc, argv, prefix,
12801307
builtin_fetch_options, builtin_fetch_usage, 0);
12811308

1309+
if (deepen_relative) {
1310+
if (deepen_relative < 0)
1311+
die(_("Negative depth in --deepen is not supported"));
1312+
if (depth)
1313+
die(_("--deepen and --depth are mutually exclusive"));
1314+
depth = xstrfmt("%d", deepen_relative);
1315+
}
12821316
if (unshallow) {
12831317
if (depth)
12841318
die(_("--depth and --unshallow cannot be used together"));
@@ -1291,6 +1325,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
12911325
/* no need to be strict, transport_set_option() will validate it again */
12921326
if (depth && atoi(depth) < 1)
12931327
die(_("depth %s is not a positive number"), depth);
1328+
if (depth || deepen_since || deepen_not.nr)
1329+
deepen = 1;
12941330

12951331
if (recurse_submodules != RECURSE_SUBMODULES_OFF) {
12961332
if (recurse_submodules_default) {

‎commit.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -267,6 +267,8 @@ extern int for_each_commit_graft(each_commit_graft_fn, void *);
267267
extern int is_repository_shallow(void);
268268
extern struct commit_list *get_shallow_commits(struct object_array *heads,
269269
int depth, int shallow_flag, int not_shallow_flag);
270+
extern struct commit_list *get_shallow_commits_by_rev_list(
271+
int ac, const char **av, int shallow_flag, int not_shallow_flag);
270272
extern void set_alternate_shallow_file(const char *path, int override);
271273
extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
272274
const struct sha1_array *extra);

0 commit comments

Comments
 (0)