Preserve mask when assigning a masked Time into an unmasked one - #20178
Preserve mask when assigning a masked Time into an unmasked one#20178Mohit-Ak wants to merge 3 commits into
Conversation
Time.__setitem__ wrote the value's jd1/jd2 straight into the target's plain ndarrays, so if the target was not yet masked the incoming mask was dropped and the masked rows showed their underlying values again. Upgrade jd1/jd2 to Masked first when the value is masked, mirroring what setting np.ma.masked already does. This is what made table vstack lose the mask on Time mixin columns: it builds the output with TimeInfo.new_like (unmasked) and then fills it via setitem. Fixes astropy#20173
|
Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.
|
|
@Mohit-Ak - first, thanks for the contribution. I pushed one commit to refactor some common code into a private method on One issue is that now This seems unavoidable when promoting to Masked, as the parent is unmasked and the view is masked. @mhvk - do you have any thoughts on the right behavior? |
|
Another issue is that a failed assignment can convert to masked but then not actually do the assignment. |
|
@taldcroft - this is quite tricky, and I don't have a ready answer. FWIW, In principle, in p.s. That |
mhvk
left a comment
There was a problem hiding this comment.
Now also looked at the actual PR: I think this looks good! Two comments, though the long first one here ends up boiling down just to a request to change the name, to _ensure_masked.
| self._time.jd2 = Masked( | ||
| self._time.jd2, mask=self._time.jd1.mask, copy=False | ||
| ) | ||
| self._time._convert_to_masked() |
There was a problem hiding this comment.
The name of the method suggests that something will always happen, but it should only do something when we're not masked, so maybe self._time._ensure_masked()?
Note that I really like that this is put to the TimeFormat class; better separation of concerns.
Actually, another suggestion: how about creating a TimeFormat.masked property which can be set? So, here, it would be self._time.masked = True (but it would not be possible to set it to False). That could then also be used inside the masked property here, separating concerns. Though I think this would be better done as follow-up; it doesn't really matter...
There was a problem hiding this comment.
And yes, let's leave the property for later.
| # If the value carries a mask but we do not, we have to upgrade our | ||
| # internal jd1/jd2 to Masked first, otherwise the mask of the value | ||
| # would be silently dropped (gh-20173). | ||
| if isinstance(value._time.jd2, Masked): |
There was a problem hiding this comment.
Here, we know value is a Time instance, so just if value.masked:
Description
Fixes #20173.
vstackwas silently dropping the mask onTimemixin columns, but the root cause turned out to sit one level down, inTime.__setitem__.The tail of
__setitem__writes the value'sjd1/jd2straight into the target's arrays:If the target
Timeis not masked yet, those are plain ndarrays. Assigning aMaskedNDArrayinto a plain ndarray keeps the data and throws the mask away, so the masked entries come back as their underlying (pre-mask) values with no error or warning.That is exactly the path
vstacktakes._vstackbuilds the output column withTimeInfo.new_like, which allocatesjd1 = np.full(shape, jd2000)/jd2 = np.zeros(shape)— deliberately unmasked, since it is meant to be filled in place — and then fills it withcol[idx0:idx1] = array[name]. The first assignment silently loses the mask.Worth noting
joinandhstackare unaffected: they index the source column directly (array[name][array_out]) instead of round-tripping throughnew_like+ setitem, so their masks survive. That is why this only showed up invstack.The fix upgrades
jd1/jd2toMaskedbefore the assignment when the incoming value is masked and we are not. This mirrors what thevalue is np.ma.maskedbranch a few lines above already does, and reuses the samemask=self._time.jd1.masksharing sojd1andjd2keep a common mask.I put the fix in
__setitem__rather than innew_likeor_vstackon purpose. Makingnew_likealways return a maskedTimewould force a mask onto every join/vstack output whether or not anything is masked, and patching_vstackalone would leave the underlying setitem hole open — plaint[:2] = masked_timeloses the mask too, independent of tables:Unmasking still behaves as before — assigning an unmasked value over a masked element clears that element's mask, and the existing
np.ma.nomaskbranch is untouched.How was this tested?
Two regression tests in
astropy/time/tests/test_mask.py:test_setitem_masked_valuecovers the setitem behaviour directly (slice assignment, scalar assignment of a masked element, sharedjd1/jd2mask, and that unmasking still works), andtest_vstack_maskedis the reporter's scenario from the issue.Both fail on
mainand pass with the fix:The reproducer in the issue now prints the expected output:
Full suites, no regressions:
Lint with the pinned ruff (
v0.15.20from.pre-commit-config.yaml):I'll add the changelog fragment once this has a PR number.