Skip to content

TST: add tests for _convolveNd_c (from convolution sub-package) C extension#19470

Open
Abu-bakkar-siddique wants to merge 8 commits intoastropy:mainfrom
Abu-bakkar-siddique:test/convolve-nd-c-extension
Open

TST: add tests for _convolveNd_c (from convolution sub-package) C extension#19470
Abu-bakkar-siddique wants to merge 8 commits intoastropy:mainfrom
Abu-bakkar-siddique:test/convolve-nd-c-extension

Conversation

@Abu-bakkar-siddique
Copy link
Copy Markdown

@Abu-bakkar-siddique Abu-bakkar-siddique commented Mar 19, 2026

Description

This PR adds direct tests for the _convolveNd_c C extension in the convolution
subpackage as part of my GSoC 2026 interest in improving low-level test coverage.

Currently no tests target this extension directly , only the public Python API (convolve.py) for this is tested.

UPDATE:

I’ve updated the file to simplify the approach:

  • removed the Python reference/oracle helper
  • replaced it with explicit expected values for small representative cases
  • renamed the helper to make the zero-padding setup explicit
  • removed the extra trimming helper and made the embed=True expectation explicit in the test
  • made dtype=np.float64 explicit throughout

Testing

pytest astropy/convolution/tests/test_extension_convolve_nd_c.py -v

cc @neutrinoceros @astrofrog @nstarman

@github-actions
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@Abu-bakkar-siddique Abu-bakkar-siddique force-pushed the test/convolve-nd-c-extension branch from de6a221 to c51851b Compare March 19, 2026 21:28
@Abu-bakkar-siddique
Copy link
Copy Markdown
Author

Should I revise the approach for the simpler contract-level cases, like kernel flipping and the bot == 0 fallback? Using the oracle there now feels a bit heavy, and explicit expected values might make those tests clearer. I still think the oracle is useful for the broader regression coverage

Copy link
Copy Markdown
Contributor

@neutrinoceros neutrinoceros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi ! thank you for tacking this. It seems like a valuable set of tests. I couldn't review everything in one sitting though, but I do have a couple points I'd like you to clarify for me, because I am not very familiar with this specific function. I also want to note I stopped my review on a particular point where I was not sure the approach was sound, as it seemed more like touching on implementation details rather than operational correctness. I'd appreciate clarifications there too. Will continue my review when you reply. Thanks again !

return result


def _trim_padding(embedded_result, kernel):
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this function seems confusing. I'll go back to it once I read the rest

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_trim_padding was there only to cut away the outer zero-padded border from the embed=True result so that the test could compare just the inner region against the non-embedded output. I removed the helper entirely and now check the expected embedded output explicitly in the test itself :

expected = np.array(
    [
        [0.0, 0.0, 0.0, 0.0, 0.0],
        [0.0, 12.0, 21.0, 16.0, 0.0],
        [0.0, 27.0, 45.0, 33.0, 0.0],
        [0.0, 24.0, 39.0, 28.0, 0.0],
        [0.0, 0.0, 0.0, 0.0, 0.0],
    ],
    dtype=np.float64,
)

return embedded_result[tuple(slices)]


# this function acts as the oracle, a reference against which the outputs of the _convolveNd_c will be validated
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've never seen the term "oracle" used in this meaning, nor could I find a reference for it. Where does this come from ?

Copy link
Copy Markdown
Author

@Abu-bakkar-siddique Abu-bakkar-siddique Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return np.zeros(shape, dtype=np.float64, order="C")


def _call_direct(image, kernel, *, nan_interpolate=False, embed=False, n_threads=1):
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the name of the function seems a bit misleading to me. It's not a direct call precisely because it contains preliminary steps too.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced it with _run_extension_with_zero_padding(...) so the name reflects the setup being done before calling the extension.

),
pytest.param(
np.arange(1.0, 65.0).reshape(4, 4, 4),
np.ones((3, 3, 3), dtype=np.float64),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this the only example input with an explicit dtype ? I think it should be specified everywhere

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I made dtype=np.float64 explicit throughout the test inputs for consistency.

npt.assert_allclose(result, expected)


def test_asymmetric_kernel_is_flipped_before_multiplication():
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you clarify, why is this behavior even expected ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original C implementation access the kernel in reverse indexing as :

 //see astropy/convolution/src/convolve.c  Line 315:

  for (ii = 0; ii < nkx; ++ii) {
      val = f[i_minus_wkx + ii];
      ker = g[nkx_minus_1 - ii];
      ...
  }

To be more clear:

ker = g[nkx_minus_1 - ii];

That is the flip. Instead of g[ii], it reads the kernel from the end toward the beginning(same pattern exists in 2D and 3D dimension).

This particular test case is to check if this is being done correctly in case of asymmetric kernel ([1, 2, 3] as kernel yield different output when flipped as [3, 2, 1]) .



def test_asymmetric_kernel_is_flipped_before_multiplication():
"""The direct extension should perform convolution, not correlation."""
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not understand how this explains the test or its name. I'd appreciate clarifications

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simply meaning that the direct extension should flip the (asymmetric) kernel, if it does not. that is not convolution (but correlation that slides the kernel without flipping)

please read my reply to the previous comment on this function for reference.

(perhaps i should have been clear about this whole idea in docstrings)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, I see that this flipping is evidently intentional within the C code, but I do not understand the nuance between convolution and correlation you're drawing. I'm absolutely not a math expert, so please educate me :) (a reference on the topic would go a long way)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out, I was under the assumption of only 2 cases kernel flipped = convolution, and kernel not flipped = correlation, but the test should be broader than that, since it is a convolution extension, mentioning correlation is probably irrelevant here, I'll rename the function and the docstring.

Copy link
Copy Markdown
Author

@Abu-bakkar-siddique Abu-bakkar-siddique Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update: I made the following changes to this function:

  • added descriptive docstring to convey test intent clearly
  • test case now checks kernel flip for 1D, 2D, and 3D
  • renamed the function to reflect the behavior being tested more accurately

please see commit a76c39f and let me know if you require any further clarifications

def test_embed_true_matches_embed_false_in_inner_region_and_leaves_border_zero(
image, kernel
):
"""Embedded results should land in the padded coordinates and nowhere else."""
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure exactly what "the padded coordinates" refers to.

Comment on lines +180 to +181
# The C loop only writes valid image coordinates, so the outer border should
# still contain the zeros from the initial result allocation.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment reads like you are exclusively writing the test against the existing behavior, instead of validating that the results are actually correct. Maybe the problem is only in the phrasing, but I would like to check with you first so we're clear what the actual approach is.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with that, reworked the tests to move away from the Python reference helper and now use explicit expected values.

@neutrinoceros neutrinoceros self-assigned this Mar 20, 2026
@neutrinoceros
Copy link
Copy Markdown
Contributor

Another note is that this is "competing" with #19455, which might be merged first because it's smaller and thus easier to review. We could very likely merge both contributions, but the second on to go in might need some additional effort to harmonize the sum.

@Abu-bakkar-siddique Abu-bakkar-siddique force-pushed the test/convolve-nd-c-extension branch from 309b52b to 36792c6 Compare March 21, 2026 15:42
@Abu-bakkar-siddique
Copy link
Copy Markdown
Author

Abu-bakkar-siddique commented Mar 21, 2026

Hi, thanks for taking the time to review this.

Looking at it again after some distance, I think your concern is valid. I leaned too much into the implementation details of convolve.c, especially with _reference_direct_convolution, and ended up writing a Python reference that mirrors the same internal assumptions and behavior rather than
testing operational correctness from the outside.

I think the better direction is to simplify these tests and rely on explicit, hard-coded expected values.

Thanks again for pointing this out , I’m going to rework the tests with that in mind.
cc @neutrinoceros

Copy link
Copy Markdown
Contributor

@neutrinoceros neutrinoceros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another pass. I still need to review the actual tests in more depth, but I'm already seeing the limits of my own understanding so I want to make sure I know what to look for. Thank you for your patience !

Comment on lines +84 to +89
result = _run_extension_with_zero_padding(
image,
kernel,
nan_interpolate=False,
embed=False,
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I meant to suggest in my previous review: it's simpler to only pass arguments that modify the default behavior (that's what default values are for)

Suggested change
result = _run_extension_with_zero_padding(
image,
kernel,
nan_interpolate=False,
embed=False,
)
result = _run_extension_with_zero_padding(image, kernel)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(please apply similar changes to all other call sites of this helper function)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! see commit e4f4a4c

],
)
def test_nan_without_interpolation_matches_expected_values(image, kernel, expected):
"""Without NaN interpolation, any NaN in the window should propagate to the output."""
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might as well also state how it's expected to propagate; the behavior you're testing is a little bit more specific than that.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a simple example in docstrings if that what you meant by how, please see commit a8c3586
If you meant something else please let me know.

pytest.param(
np.ones((3, 3, 3), dtype=np.float64),
np.ones((3, 3, 3), dtype=np.float64),
np.full((3, 3, 3), np.nan, dtype=np.float64),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a double take here. This is misleading and relies on the test's body to modify the input.
I think it's actually preferable to hard code the actual input here.
You'll probably want to disable auto-formatting locally for a 3D literal array

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done ! please see commit 975b0ee.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also ruff formatter would not allow to diable autoformating just for the 3D literals, nor for the @pytest.mark.parametrize decorator, so had to wrap the whole function (after formating it correctly)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants