reftable, ewah: fix bounds checks, record comparisons, and indexed refs by inanna-malick · Pull Request #2477 · git/git

inanna-malick · 2026-10-30T05:49:13Z

Fix memory safety and ordering bugs in reftable/ and ewah/:

In reftable_obj_record_cmp_void() (reftable/record.c), pass the smaller of a->hash_prefix_len and b->hash_prefix_len to memcmp() instead of the larger length so comparing object records of different prefix lengths does not read past the shorter hash_prefix buffer.
In reftable_log_record_cmp_void() (reftable/record.c), compare uint64_t update_index values directly instead of returning b->update_index - a->update_index, which truncates the 64-bit difference to int.
In reftable_block_init() (reftable/block.c), check that block->block_data.len >= header_size + 4 before reading block_type and the 24-bit block_size at data + header_size.
In reftable_table_refs_for_indexed() (reftable/table.c), verify got.u.obj.hash_prefix_len == t->object_id_len before comparing t->object_id_len bytes against got.u.obj.hash_prefix.
In indexed_table_ref_iter_next() (reftable/iter.c), check ref->value_type (REFTABLE_REF_VAL1 vs REFTABLE_REF_VAL2) before reading ref->value.val2 so REFTABLE_REF_DELETION and REFTABLE_REF_SYMREF records are not matched against union bytes, and add it->table->min_update_index to ref->update_index consistently with table_iter_next_in_block().
In ewah/bitmap.c, ewah/ewah_io.c, ewah/ewah_bitmap.c, and ewah/ewok_rlw.h, validate rlw offset and literal word bounds against self->buffer_size, grow self->words in bitmap_or_ewah() when compressed words exceed other->bit_size, and store RLW counts as size_t in struct rlw_iterator to avoid 32-bit signed int sign-extension.

Found via agent swarm driven property tests using https://recursion.wtf/posts/agents-and-property-tests/

gitgitgadget-git · 2026-10-30T05:49:48Z

Welcome to GitGitGadget

Hi @inanna-malick, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests.

Please make sure that either:

Your Pull Request has a good description, if it consists of multiple commits, as it will be used as cover letter.
Your Pull Request description is empty, if it consists of a single commit, as the commit message should be descriptive enough by itself.

You can CC potential reviewers by adding a footer to the PR description with the following syntax:

CC: Revi Ewer <revi.ewer@example.com>, Ill Takalook <ill.takalook@example.net>

NOTE: DO NOT copy/paste your CC list from a previous GGG PR's description,
because it will result in a malformed CC list on the mailing list. See
example.

Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:

the lines should not exceed 76 columns,
the first line should be like a header and typically start with a prefix like "tests:" or "revisions:" to state which subsystem the change is about, and
the commit messages' body should be describing the "why?" of the change.
Finally, the commit messages should end in a Signed-off-by: line matching the commits' author.

It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code.

Contributing the patches

Before you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form /allow. A good way to find other contributors is to locate recent pull requests where someone has been /allowed:

Search: is:pr is:open "/allow"

Both the person who commented /allow and the PR author are able to /allow you.

An alternative is the channel #git-devel on the Libera Chat IRC network:

<newcontributor> I've just created my first PR, could someone please /allow me? https://github.com/gitgitgadget/git/pull/12345
<veteran> newcontributor: it is done
<newcontributor> thanks!

Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment /submit.

If you want to see what email(s) would be sent for a /submit request, add a PR comment /preview to have the email(s) sent to you. You must have a public GitHub email address for this. Note that any reviewers CC'd via the list in the PR description will not actually be sent emails.

After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail).

If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the (raw) link), then import it into your mail program. If you use GMail, you can do this via:

curl -g --user "<EMailAddress>:<Password>" \
    --url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txt

To iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description):

Changes since v1:
- Fixed a typo in the commit message (found by ...)
- Added a code comment to ... as suggested by ...
...

To send a new iteration, just add another PR comment with the contents: /submit.

Need help?

New contributors who want advice are encouraged to join git-mentoring@googlegroups.com, where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join.

You may also be able to find help in real time in the developer IRC channel, #git-devel on Libera Chat. Remember that IRC does not support offline messaging, so if you send someone a private message and log out, they cannot respond to you. The scrollback of #git-devel is archived, though.

Copilot

🟡 Changes recommended

Bitmap decoding can trigger quadratic copying through per-word reallocations, and regression tests do not adequately cover the iterator and record-comparison fixes.

4 open issues

What changed in this PR

This PR addresses memory-safety and ordering bugs in Git’s reftable storage and EWAH bitmap code, and corrects indexed reference lookup behavior.

Changes:

Add bounds checks for reftable headers, object prefixes, and EWAH buffer access.
Correct record comparisons and indexed reference filtering and update indexes.
Grow bitmap storage when decoded words exceed the advertised size and use size_t for run-length iterator counts.

File	Description
reftable/table.c	Validate object-prefix length before comparison.
reftable/record.c	Bound prefix comparisons and preserve 64-bit update-index ordering.
reftable/iter.c	Filter references by value type and restore update indexes.
reftable/block.c	Check available bytes before reading block headers.
ewah/ewok_rlw.h	Use `size_t` for iterator counts and offsets.
ewah/ewah_io.c	Reject running-length word offsets outside the buffer.
ewah/ewah_bitmap.c	Stop literal-word reads at the buffer boundary.
ewah/bitmap.c	Expand destination storage for excess decoded words.

_{Review effort: Balanced}

+		if (i >= self->word_alloc) {
+			size_t old_size = self->word_alloc;
+			self->word_alloc = st_add(i, 1);
+			REALLOC_ARRAY(self->words, self->word_alloc);
+			MEMZERO_ARRAY(self->words + old_size,
+				      self->word_alloc - old_size);
+		}


+		ref->update_index += it->table->min_update_index;
+		if (ref->value_type == REFTABLE_REF_VAL2 &&
+		    (!memcmp(it->oid.buf, ref->value.val2.target_value,
+			     it->oid.len) ||
+		     !memcmp(it->oid.buf, ref->value.val2.value,


-	cmp = memcmp(a->hash_prefix, b->hash_prefix,
-		     a->hash_prefix_len > b->hash_prefix_len ?
-		     a->hash_prefix_len : b->hash_prefix_len);
+	cmp = memcmp(a->hash_prefix, b->hash_prefix, min_len);


+	if (a->update_index != b->update_index)
+		return a->update_index > b->update_index ? -1 : 1;
+	return 0;


Fix memory safety and ordering bugs in reftable/ and ewah/: 1. In reftable_obj_record_cmp_void() (reftable/record.c), pass the smaller of `a->hash_prefix_len` and `b->hash_prefix_len` to `memcmp()` instead of the larger length so comparing object records of different prefix lengths does not read past the shorter `hash_prefix` buffer. 2. In reftable_log_record_cmp_void() (reftable/record.c), compare `uint64_t` `update_index` values directly instead of returning `b->update_index - a->update_index`, which truncates the 64-bit difference to `int`. 3. In reftable_block_init() (reftable/block.c), check that `block->block_data.len >= header_size + 4` before reading `block_type` and the 24-bit `block_size` at `data + header_size`. 4. In reftable_table_refs_for_indexed() (reftable/table.c), verify `got.u.obj.hash_prefix_len == t->object_id_len` before comparing `t->object_id_len` bytes against `got.u.obj.hash_prefix`. 5. In indexed_table_ref_iter_next() (reftable/iter.c), check `ref->value_type` (`REFTABLE_REF_VAL1` vs `REFTABLE_REF_VAL2`) before reading `ref->value.val2` so `REFTABLE_REF_DELETION` and `REFTABLE_REF_SYMREF` records are not matched against union bytes, and add `it->table->min_update_index` to `ref->update_index` consistently with `table_iter_next_in_block()`. 6. In `ewah/bitmap.c`, `ewah/ewah_io.c`, `ewah/ewah_bitmap.c`, and `ewah/ewok_rlw.h`, validate `rlw` offset and literal word bounds against `self->buffer_size`, grow `self->words` in `bitmap_or_ewah()` when compressed words exceed `other->bit_size`, and store RLW counts as `size_t` in `struct rlw_iterator` to avoid 32-bit signed int sign-extension. Found via agent swarm driven property tests using https://recursion.wtf/posts/agents-and-property-tests/ Signed-off-by: Inanna Malick <inanna@recursion.wtf>

carenas · 2026-10-30T11:11:37Z

/allow

gitgitgadget-git · 2026-10-30T11:12:16Z

User inanna-malick already allowed to use GitGitGadget.

Copilot AI balanced review requested due to automatic review settings October 30, 2026 05:49

github-project-automation Bot added this to Op-Amp Frequency Response and Applications Suite Oct 30, 2026

github-project-automation Bot moved this to Todo in Op-Amp Frequency Response and Applications Suite Oct 30, 2026

Copilot started reviewing on behalf of inanna-malick October 30, 2026 05:49

View session

gitgitgadget-git Bot added the new user label Oct 30, 2026

Copilot AI reviewed Oct 30, 2026

View reviewed changes

inanna-malick force-pushed the fix/reftable-and-ewah-bounds-checks branch from cd3f728 to 92fd9f0 Compare October 30, 2026 05:55

Navigation Menu

Repository navigation

reftable, ewah: fix bounds checks, record comparisons, and indexed refs - #2477

reftable, ewah: fix bounds checks, record comparisons, and indexed refs#2477
inanna-malick wants to merge 1 commit into
git:masterfrom
inanna-malick:fix/reftable-and-ewah-bounds-checks

inanna-malick commented Oct 30, 2026

Uh oh!

gitgitgadget-git Bot commented Oct 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

carenas commented Oct 30, 2026

Uh oh!

gitgitgadget-git Bot commented Oct 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

inanna-malick commented Oct 30, 2026

Uh oh!

gitgitgadget-git Bot commented Oct 30, 2026

Welcome to GitGitGadget

Contributing the patches

Need help?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

carenas commented Oct 30, 2026

Uh oh!

gitgitgadget-git Bot commented Oct 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants