Support reftable ref backend for Git by hanwen · Pull Request #1054 · git/git

hanwen · 2021-07-19T17:50:41Z

This continues the work in #847, which the gitgitgadget erroneously closed.

Changes relative to last series (version 17 Aug 2021, tip: 3110d6e)

drop FUNCTION use
use REALLOC_ARRAY; remove informational errno
avarab's comments on tests
fix assert failure in worktree reflog expiry.

For the 'seen' branch, the following should be applied:

diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index d7137d1213..9323931eeb 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -108,7 +108,7 @@ static const char *bare_ref_name(const char *ref)
 static int git_reftable_read_raw_ref(struct ref_store *ref_store,
 				     const char *refname, struct object_id *oid,
 				     struct strbuf *referent,
-				     unsigned int *type);
+				     unsigned int *type, int *failure_errno);
 
 static void clear_reftable_log_record(struct reftable_log_record *log)
 {
@@ -425,13 +424,14 @@ static int fixup_symrefs(struct ref_store *ref_store,
 	for (i = 0; i < transaction->nr; i++) {
 		struct ref_update *update = transaction->updates[i];
 		struct object_id old_oid;
+		int failure_errno;
 
 		err = git_reftable_read_raw_ref(ref_store, update->refname,
 						&old_oid, &referent,
 						/* mutate input, like
 						   files-backend.c */
-						&update->type);
-		if (err < 0 && errno == ENOENT &&
+						&update->type, &failure_errno);
+		if (err < 0 && failure_errno == ENOENT &&
 		    is_null_oid(&update->old_oid)) {
 			err = 0;
 		}
@@ -1602,7 +1603,7 @@ static int reftable_error_to_errno(int err)
 static int git_reftable_read_raw_ref(struct ref_store *ref_store,
 				     const char *refname, struct object_id *oid,
 				     struct strbuf *referent,
-				     unsigned int *type)
+				     unsigned int *type, int *failure_errno)
 {
 	struct git_reftable_ref_store *refs =
 		(struct git_reftable_ref_store *)ref_store;
@@ -1626,13 +1627,11 @@ static int git_reftable_read_raw_ref(struct ref_store *ref_store,
 
 	err = reftable_stack_read_ref(stack, refname, &ref);
 	if (err > 0) {
-		errno = ENOENT;
+		*failure_errno = ENOENT;
 		err = -1;
 		goto done;
 	}
 	if (err < 0) {
-		errno = reftable_error_to_errno(err);
-		err = -1;
 		goto done;
 	}

cc: Carlo Marcelo Arenas Belón carenas@gmail.com
cc: Han-Wen Nienhuys hanwen@google.com
cc: Philip Oakley philipoakley@iee.email
cc: Ævar Arnfjörð Bjarmason avarab@gmail.com
cc: Carlo Arenas carenas@gmail.com

cc: Han-Wen Nienhuys hanwen@google.com
cc: Ævar Arnfjörð Bjarmason avarab@gmail.com

hanwen · 2021-07-20T17:03:57Z

/submit

gitgitgadget-git · 2021-07-20T17:05:19Z

Submitted as pull.1054.git.git.1626800686.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git pr-git-1054/hanwen/libreftable-v1

To fetch this version to local tag pr-git-1054/hanwen/libreftable-v1:

git fetch --no-tags https://github.com/gitgitgadget/git tag pr-git-1054/hanwen/libreftable-v1

gitgitgadget-git · 2021-07-20T21:23:25Z

This branch is now known as hn/reftable.

gitgitgadget-git · 2021-07-20T21:23:26Z

This patch series was integrated into seen via f55539d.

gitgitgadget-git · 2021-07-20T23:20:11Z

This patch series was integrated into seen via 57102f2.

gitgitgadget-git · 2021-07-21T18:38:54Z

This patch series was integrated into seen via bd46e67.

gitgitgadget-git · 2021-07-21T23:55:52Z

This patch series was integrated into seen via 45f92dd.

gitgitgadget-git · 2021-07-22T02:29:09Z

There was a status update in the "Cooking" section about the branch hn/reftable on the Git mailing list:

The "reftable" backend for the refs API.

gitgitgadget-git · 2021-07-22T22:07:59Z

This patch series was integrated into seen via 1871e2f.

gitgitgadget-git · 2021-07-24T00:41:37Z

This patch series was integrated into seen via 7cfeb34.

gitgitgadget-git · 2021-07-27T00:13:06Z

This patch series was integrated into seen via d80dfcd.

gitgitgadget-git · 2021-07-27T00:41:53Z

This patch series was integrated into seen via bdce115.

gitgitgadget-git · 2021-07-27T04:59:59Z

There was a status update in the "Cooking" section about the branch hn/reftable on the Git mailing list:

The "reftable" backend for the refs API.

gitgitgadget-git · 2021-07-28T01:20:58Z

This patch series was integrated into seen via dbf640e.

gitgitgadget-git · 2021-07-28T05:31:55Z

This patch series was integrated into seen via e5c19a9.

gitgitgadget-git · 2021-07-28T22:17:37Z

This patch series was integrated into seen via 94646ba.

gitgitgadget-git · 2021-07-29T00:16:40Z

There was a status update in the "Cooking" section about the branch hn/reftable on the Git mailing list:

The "reftable" backend for the refs API.

Seems to break CI jobs in 'seen'.

gitgitgadget-git · 2021-07-29T23:34:18Z

This patch series was integrated into seen via fad749e.

gitgitgadget-git · 2021-07-30T21:52:47Z

This patch series was integrated into seen via 13ec157.

gitgitgadget-git · 2021-08-02T03:47:12Z

This patch series was integrated into seen via 2d869d8.

gitgitgadget-git · 2021-08-02T05:47:20Z

There was a status update in the "Cooking" section about the branch hn/reftable on the Git mailing list:

The "reftable" backend for the refs API.

Seems to break CI jobs in 'seen'.

gitgitgadget-git · 2021-08-02T23:05:30Z

This patch series was integrated into seen via c40aa07.

gitgitgadget-git · 2021-08-04T01:42:16Z

This patch series was integrated into seen via 3a7adf1.

gitgitgadget-git · 2021-08-04T07:18:14Z

There was a status update in the "Cooking" section about the branch hn/reftable on the Git mailing list:

The "reftable" backend for the refs API.

Seems to break CI jobs in 'seen'.

gitgitgadget-git · 2021-08-06T00:11:17Z

This patch series was integrated into seen via 22ab9da.

gitgitgadget-git · 2021-08-07T00:21:14Z

There was a status update in the "Cooking" section about the branch hn/reftable on the Git mailing list:

The "reftable" backend for the refs API.

Seems to break CI jobs in 'seen'.
cf. https://github.com/git/git/runs/3257622953?check_suite_focus=3Dtrue#=
step:4:1685

gitgitgadget-git · 2021-08-09T19:26:47Z

This patch series was integrated into seen via a15fa07.

gitgitgadget-git · 2021-08-11T01:15:45Z

There was a status update in the "Cooking" section about the branch hn/reftable on the Git mailing list:

The "reftable" backend for the refs API.

Seems to break CI jobs in 'seen'.
cf. https://github.com/git/git/runs/3257622953?check_suite_focus=3Dtrue#=
step:4:1685

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

provide a command-line utility for inspecting individual tables, and inspecting a complete ref database Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Helped-by: Carlo Marcelo Arenas Belón <carenas@gmail.com>

This command dumps individual tables or a stack of of tables. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

The reftable backend needs to know the hash algorithm for writing the initialization hash table. The initial reftable contains a symref HEAD => "main" (or "master"), which is agnostic to the size of hash value, but this is an exceptional circumstance, and the reftable library does not cater to this exception. It insists that all tables in the stack have a consistent format ID for the hash algorithm. Call set_repo_hash_algo directly after calling validate_hash_algorithm() (which reads $GIT_DEFAULT_HASH). Helped-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

gitgitgadget-git · 2021-08-30T14:09:26Z

On the Git mailing list, Ævar Arnfjörð Bjarmason wrote (reply to this):

On Mon, Aug 30 2021, Han-Wen Nienhuys wrote:

> On Mon, Aug 30, 2021 at 3:22 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>> But we will need at least the optimistic locking of code like
>> builtin/reflog.c wanting to do an expiry, and deciding whether to do
>> that expiry based on a given state of the ref/reflog. I.e. we don't
>> want:
>>
>>     1. Start reflog expiry
>>     2. Code in builtin/reflog.c looks up the OID
>>     3. Code in builtin/reflog.c decides whether expire the reflog
>>     4. Concurrent with #4, another writer updates the ref/reflog pair
>>     5. Code in builtin/reflog.c says "OK, expire it!"
>>     6. Reftable queues a delete/prune of the reflog per #5.
>>
>> This would be a sequente of updates to the ref/reflog, none of whom were
>> racy as far as the reftable semantics itself are concerneb, but where
>> we'd do the wrong thing because the writer thought we had A when we
>> really had B. So we need the equivalent of an "git update-ref" with the
>> "<oldvalue>".
>>
>> Is there a better way to do that in this case that I'm missing?
>
> I spent some more time looking at builtin/reflog.c, but I am still not
> 100% sure what the locking is used for.
>
> From a quick glance, the OID goes into tip_commit, and tip_commit goes
> into a reachable list (?). The reachable list is then for something,
> but I can't really tell what.
>
> In your example with 1.-6., it's still not clear to me what the
> undesired behavior is precisely. If the reflog is pruned in #6, is the
> worry that the update in #4 is pruned immediately after being
> effected?

Yes, I think so. But I'm not sure. I skimmed the code quickly today, and
when I wrote the referenced series didn't focus much on the nitty-gritty
of the builtin/reflog.c behavior other than assuring myself that I was
doing the exact same thing as before as far as its logic was concerned.

I.e. it always locked at a given OID. Before my in-flight "reflog
expire: don't lock reflogs using previously seen OI" it might not lock
but get this error:

    error: cannot lock ref '<refname>': ref '<refname>' is at <OID-A> but expected <OID-B>

But at least it wouldn't do anything, but the current code does require
the passed-in OID. See the code that needs "unreachable_expire_kind" and
"tip_commit".

Perhaps that whole thing can also be refactored somehow. If I change the
"commit = lookup_commit(the_repository, oid);" in
"reflog_expiry_prepare()" to just "commit = NULL" all tests pass, but
that might just be missing test coverage in the face of same race...

gitgitgadget-git · 2021-08-30T14:09:28Z

User Ævar Arnfjörð Bjarmason <avarab@gmail.com> has been added to the cc: list.

For background, see Documentation/technical/reftable.txt. This introduces the file refs/reftable-backend.c containing a reftable-powered ref storage backend. It can be activated by setting GIT_TEST_REFTABLE in the environment. When GIT_TEST_REFTABLE is set, the test prerequisite !REFFILES is set. There is no option to git-init for now, as the test suite still shows failures with GIT_TEST_REFTABLE=1. Example use: see t/t0031-reftable.sh Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Helped-by: Junio Hamano <gitster@pobox.com> Helped-by: Patrick Steinhardt <patrick.steinhardt@elego.de> Co-authored-by: Jeff King <peff@peff.net>

In our git-prompt script we strive to use Bash builtins wherever possible, because fork()-ing subshells for command substitutions and fork()+exec()-ing Git commands are expensive on some platforms. We even read and parse '.git/HEAD' using Bash builtins to get the name of the current branch [1]. However, the upcoming reftable refs backend won't use '.git/HEAD' at all, but will write an invalid refname as placeholder for backwards compatibility instead, which will break our git-prompt script. Update the git-prompt script to recognize the placeholder '.git/HEAD' written by the reftable backend (its content is specified in the reftable specs), and then fall back to use 'git symbolic-ref' to get the name of the current branch. [1] 3a43c4b (bash prompt: use bash builtins to find out current branch, 2011-03-31) Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

* Reftable for now lacks detailed error messages for directory/file conflicts. Skip message comparisons. * Mark tests that muck with .git directly as REFFILES. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

gitgitgadget-git · 2021-08-30T15:32:18Z

This patch series was integrated into seen via 364cf33.

gitgitgadget-git · 2021-08-31T00:05:56Z

This patch series was integrated into seen via fa30190.

gitgitgadget-git · 2021-08-31T06:15:13Z

This patch series was integrated into seen via ee9e84b.

gitgitgadget-git · 2021-08-31T15:13:58Z

This patch series was integrated into seen via 8a94a63.

gitgitgadget-git · 2021-08-31T21:55:54Z

This patch series was integrated into seen via 3d89fcd.

gitgitgadget-git · 2021-09-01T00:13:52Z

There was a status update in the "Cooking" section about the branch hn/reftable on the Git mailing list:

The "reftable" backend for the refs API.

gitgitgadget-git · 2021-09-01T22:25:33Z

This patch series was integrated into seen via d1bad54.

gitgitgadget-git · 2021-09-01T23:43:32Z

This patch series was integrated into seen via 11bad4c.

gitgitgadget-git · 2021-09-02T01:11:30Z

This patch series was integrated into seen via 4e18755.

gitgitgadget-git · 2021-09-02T21:33:34Z

This patch series was integrated into seen via d009cb5.

gitgitgadget-git · 2021-09-02T22:53:14Z

There was a status update in the "Cooking" section about the branch hn/reftable on the Git mailing list:

The "reftable" backend for the refs API.

gitgitgadget-git · 2021-09-03T21:51:37Z

This patch series was integrated into seen via c5d5360.

gitgitgadget-git · 2021-09-03T22:37:54Z

This patch series was integrated into seen via abe2dc2.

gitgitgadget-git · 2021-09-06T06:18:05Z

This patch series was integrated into seen via a2ede59.

gitgitgadget-git · 2021-09-06T07:00:32Z

This patch series was integrated into seen via d4f001f.

gitgitgadget-git · 2021-12-16T02:37:06Z

There was a status update in the "Graduated to 'master'" section about the branch hn/reftable on the Git mailing list:

The "reftable" backend for the refs API, without integrating into
the refs subsystem, has been added.
source: <pull.1081.v4.git.git.1633638315.gitgitgadget@gmail.com>

hanwen · 2022-10-24T15:45:34Z

closing this; the library has been merged, support in Git itself is tracked in PR #1215

hanwen changed the title ~~Libreftable~~ Support reftable ref backend for Git Jul 19, 2021

gitgitgadget-git bot added the seen label Jul 20, 2021

hanwen added 4 commits August 30, 2021 16:00

reftable: implement stack, a mutable database of reftable files.

a7ec1d3

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

reftable: add dump utility

85b56c1

provide a command-line utility for inspecting individual tables, and inspecting a complete ref database Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Helped-by: Carlo Marcelo Arenas Belón <carenas@gmail.com>

Add "test-tool dump-reftable" command.

9e54004

This command dumps individual tables or a stack of of tables. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

hanwen and others added 5 commits August 30, 2021 16:27

t1301: document what needs to be done for reftable

9fd1aad

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

t1401,t2011: parameterize HEAD.lock for REFFILES

45c09cd

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

t1404: annotate test cases with REFFILES

611abb7

* Reftable for now lacks detailed error messages for directory/file conflicts. Skip message comparisons. * Mark tests that muck with .git directly as REFFILES. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

dscho removed the seen label Feb 15, 2022

hanwen closed this Oct 24, 2022

Conversation

hanwen commented Jul 19, 2021 • edited by gitgitgadget-git bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hanwen commented Jul 20, 2021

Uh oh!

gitgitgadget-git bot commented Jul 20, 2021

Uh oh!

gitgitgadget-git bot commented Jul 20, 2021

Uh oh!

gitgitgadget-git bot commented Jul 20, 2021

Uh oh!

gitgitgadget-git bot commented Jul 20, 2021

Uh oh!

gitgitgadget-git bot commented Jul 21, 2021

Uh oh!

gitgitgadget-git bot commented Jul 21, 2021

Uh oh!

gitgitgadget-git bot commented Jul 22, 2021

Uh oh!

gitgitgadget-git bot commented Jul 22, 2021

Uh oh!

gitgitgadget-git bot commented Jul 24, 2021

Uh oh!

gitgitgadget-git bot commented Jul 27, 2021

Uh oh!

gitgitgadget-git bot commented Jul 27, 2021

Uh oh!

gitgitgadget-git bot commented Jul 27, 2021

Uh oh!

gitgitgadget-git bot commented Jul 28, 2021

Uh oh!

gitgitgadget-git bot commented Jul 28, 2021

Uh oh!

gitgitgadget-git bot commented Jul 28, 2021

Uh oh!

gitgitgadget-git bot commented Jul 29, 2021

Uh oh!

gitgitgadget-git bot commented Jul 29, 2021

Uh oh!

gitgitgadget-git bot commented Jul 30, 2021

Uh oh!

gitgitgadget-git bot commented Aug 2, 2021

Uh oh!

gitgitgadget-git bot commented Aug 2, 2021

Uh oh!

gitgitgadget-git bot commented Aug 2, 2021

Uh oh!

gitgitgadget-git bot commented Aug 4, 2021

Uh oh!

gitgitgadget-git bot commented Aug 4, 2021

Uh oh!

gitgitgadget-git bot commented Aug 6, 2021

Uh oh!

gitgitgadget-git bot commented Aug 7, 2021

Uh oh!

gitgitgadget-git bot commented Aug 9, 2021

Uh oh!

gitgitgadget-git bot commented Aug 11, 2021

Uh oh!

gitgitgadget-git bot commented Aug 30, 2021

Uh oh!

gitgitgadget-git bot commented Aug 30, 2021

Uh oh!

gitgitgadget-git bot commented Aug 30, 2021

Uh oh!

gitgitgadget-git bot commented Aug 31, 2021

Uh oh!

gitgitgadget-git bot commented Aug 31, 2021

Uh oh!

gitgitgadget-git bot commented Aug 31, 2021

Uh oh!

gitgitgadget-git bot commented Aug 31, 2021

Uh oh!

gitgitgadget-git bot commented Sep 1, 2021

Uh oh!

gitgitgadget-git bot commented Sep 1, 2021

Uh oh!

gitgitgadget-git bot commented Sep 1, 2021

Uh oh!

hanwen commented Jul 19, 2021 •

edited by gitgitgadget-git bot

Loading