Skip to content

Preserve mask when assigning a masked Time into an unmasked one - #20178

Open
Mohit-Ak wants to merge 3 commits into
astropy:mainfrom
Mohit-Ak:fix/time-setitem-masked-value
Open

Preserve mask when assigning a masked Time into an unmasked one#20178
Mohit-Ak wants to merge 3 commits into
astropy:mainfrom
Mohit-Ak:fix/time-setitem-masked-value

Conversation

@Mohit-Ak

Copy link
Copy Markdown

Description

Fixes #20173.

vstack was silently dropping the mask on Time mixin columns, but the root cause turned out to sit one level down, in Time.__setitem__.

The tail of __setitem__ writes the value's jd1/jd2 straight into the target's arrays:

self._time.jd1[item] = value._time.jd1
self._time.jd2[item] = value._time.jd2

If the target Time is not masked yet, those are plain ndarrays. Assigning a MaskedNDArray into a plain ndarray keeps the data and throws the mask away, so the masked entries come back as their underlying (pre-mask) values with no error or warning.

That is exactly the path vstack takes. _vstack builds the output column with TimeInfo.new_like, which allocates jd1 = np.full(shape, jd2000) / jd2 = np.zeros(shape) — deliberately unmasked, since it is meant to be filled in place — and then fills it with col[idx0:idx1] = array[name]. The first assignment silently loses the mask.

Worth noting join and hstack are unaffected: they index the source column directly (array[name][array_out]) instead of round-tripping through new_like + setitem, so their masks survive. That is why this only showed up in vstack.

The fix upgrades jd1/jd2 to Masked before the assignment when the incoming value is masked and we are not. This mirrors what the value is np.ma.masked branch a few lines above already does, and reuses the same mask=self._time.jd1.mask sharing so jd1 and jd2 keep a common mask.

I put the fix in __setitem__ rather than in new_like or _vstack on purpose. Making new_like always return a masked Time would force a mask onto every join/vstack output whether or not anything is masked, and patching _vstack alone would leave the underlying setitem hole open — plain t[:2] = masked_time loses the mask too, independent of tables:

t = Time(["2000:001", "2000:002", "2000:003"])
value = Time(["2001:001", "2001:002"])
value[1] = np.ma.masked
t[:2] = value
t.mask  # [False, False, False] before, [False, True, False] after

Unmasking still behaves as before — assigning an unmasked value over a masked element clears that element's mask, and the existing np.ma.nomask branch is untouched.

How was this tested?

Two regression tests in astropy/time/tests/test_mask.py: test_setitem_masked_value covers the setitem behaviour directly (slice assignment, scalar assignment of a masked element, shared jd1/jd2 mask, and that unmasking still works), and test_vstack_masked is the reporter's scenario from the issue.

Both fail on main and pass with the fix:

$ python -m pytest astropy/time/tests/test_mask.py::test_setitem_masked_value \
                   astropy/time/tests/test_mask.py::test_vstack_masked -q
# before: 2 failed  (assert t.masked -> AssertionError: assert False)
# after:  2 passed

The reproducer in the issue now prints the expected output:

combined mask: [False  True  True False]

Full suites, no regressions:

$ python -m pytest astropy/time/       1002 passed, 30 skipped, 9 xfailed
$ python -m pytest astropy/table/      2407 passed, 189 skipped, 14 xfailed
$ python -m pytest astropy/utils/masked/ astropy/timeseries/ astropy/io/misc/
                                       3845 passed, 269 skipped, 2 xfailed

Lint with the pinned ruff (v0.15.20 from .pre-commit-config.yaml):

$ ruff check astropy/time/core.py astropy/time/tests/test_mask.py
All checks passed!
$ ruff format --check astropy/time/core.py astropy/time/tests/test_mask.py
2 files already formatted

I'll add the changelog fragment once this has a PR number.

Mohit-Ak added 2 commits July 31, 2026 02:17
Time.__setitem__ wrote the value's jd1/jd2 straight into the target's
plain ndarrays, so if the target was not yet masked the incoming mask
was dropped and the masked rows showed their underlying values again.
Upgrade jd1/jd2 to Masked first when the value is masked, mirroring what
setting np.ma.masked already does.

This is what made table vstack lose the mask on Time mixin columns: it
builds the output with TimeInfo.new_like (unmasked) and then fills it
via setitem.

Fixes astropy#20173
@github-actions github-actions Bot added the time label Jul 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@pllim pllim added this to the v8.1.0 milestone Jul 31, 2026
@Mohit-Ak
Mohit-Ak marked this pull request as ready for review August 10, 2026 14:12
@Mohit-Ak
Mohit-Ak requested a review from taldcroft as a code owner August 10, 2026 14:12
@taldcroft

taldcroft commented Aug 10, 2026

Copy link
Copy Markdown
Member

@Mohit-Ak - first, thanks for the contribution. I pushed one commit to refactor some common code into a private method on TimeFormat.

One issue is that now Masked(jd1, copy=False) shares the data buffer but allocates a fresh mask array, so a view's mask never reaches the parent.

import numpy as np
from astropy.time import Time

t = Time(["2001:001", "2001:002", "2001:003", "2001:004"])
s = t[1:3]                                    # view sharing t's jd1/jd2 buffers
print(np.shares_memory(t._time.jd1, s._time.jd1))   # True

masked_value = Time(["2010:001", "2010:002"])
masked_value[1] = np.ma.masked

s[:] = masked_value

print(s.mask)      # [False  True]
print(t.masked)    # False
print(t.value[2])  # 2010:002:00:00:00.000   <-- should be hidden

This seems unavoidable when promoting to Masked, as the parent is unmasked and the view is masked.

@mhvk - do you have any thoughts on the right behavior?

@taldcroft

Copy link
Copy Markdown
Member

Another issue is that a failed assignment can convert to masked but then not actually do the assignment.

import io, numpy as np
from astropy.table import QTable
from astropy.time import Time

t = Time(["2001:001", "2001:002", "2001:003", "2001:004"])
v = Time(["2010:001", "2010:002"]); v[1] = np.ma.masked

print(t.masked)                # False
try:
    t[:3] = v                  # shape mismatch
except ValueError as exc:
    print(exc)                 # could not broadcast input array from shape (2,) into shape (3,)
print(t.masked, t.mask)        # True [False False False False]

@mhvk

mhvk commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

@taldcroft - this is quite tricky, and I don't have a ready answer. FWIW, np.ma.MaskedArray(array) works the same way: if you set an element to something masked, the value in array will change too, but it will of course not get masked.

In principle, in Time, we do have information on whether we are a slice or own our own data, so in that sense we can protect users by, e.g., not allowing a view to become masked (or warning or whatever); see the end of _apply and the use of _id_cache to ensure that caches get cleared if something is written too.

p.s. That Time becomes masked and an assignment then fails is clearly a bug. Thankfully, one that is not too difficult to fix.

@mhvk mhvk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now also looked at the actual PR: I think this looks good! Two comments, though the long first one here ends up boiling down just to a request to change the name, to _ensure_masked.

Comment thread astropy/time/core.py
self._time.jd2 = Masked(
self._time.jd2, mask=self._time.jd1.mask, copy=False
)
self._time._convert_to_masked()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name of the method suggests that something will always happen, but it should only do something when we're not masked, so maybe self._time._ensure_masked()?

Note that I really like that this is put to the TimeFormat class; better separation of concerns.

Actually, another suggestion: how about creating a TimeFormat.masked property which can be set? So, here, it would be self._time.masked = True (but it would not be possible to set it to False). That could then also be used inside the masked property here, separating concerns. Though I think this would be better done as follow-up; it doesn't really matter...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the name.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And yes, let's leave the property for later.

Comment thread astropy/time/core.py
# If the value carries a mask but we do not, we have to upgrade our
# internal jd1/jd2 to Masked first, otherwise the mask of the value
# would be silently dropped (gh-20173).
if isinstance(value._time.jd2, Masked):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, we know value is a Time instance, so just if value.masked:

@taldcroft taldcroft modified the milestones: v8.1.0, v7.2.3 Aug 11, 2026
@taldcroft taldcroft added backport-v7.2.x on-merge: backport to v7.2.x backport-v8.0.x on-merge: backport to v8.0.x labels Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-v7.2.x on-merge: backport to v7.2.x backport-v8.0.x on-merge: backport to v8.0.x time

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vstack silently drops the mask on a masked Time mixin column, restoring the masked value

4 participants