Skip to content

Pandas NaT (not a time) not supported #4000

Closed

Description

@jvspl

Trying to plot the NaT (not a time) value of pandas gives an error. Feature request: treat NaT just like NaN from numpy.

from pylab import *
import pandas as pd
plot(pd.NaT,5)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-60addc339c2f> in <module>()
----> 1 plot(pd.NaT,5)

C:\Anaconda\lib\site-packages\matplotlib\pyplot.pyc in plot(*args, **kwargs)
   3091         ax.hold(hold)
   3092     try:
-> 3093         ret = ax.plot(*args, **kwargs)
   3094         draw_if_interactive()
   3095     finally:

C:\Anaconda\lib\site-packages\matplotlib\axes\_axes.pyc in plot(self, *args, **kwargs)
   1372 
   1373         for line in self._get_lines(*args, **kwargs):
-> 1374             self.add_line(line)
   1375             lines.append(line)
   1376 

C:\Anaconda\lib\site-packages\matplotlib\axes\_base.pyc in add_line(self, line)
   1484             line.set_clip_path(self.patch)
   1485 
-> 1486         self._update_line_limits(line)
   1487         if not line.get_label():
   1488             line.set_label('_line%d' % len(self.lines))

C:\Anaconda\lib\site-packages\matplotlib\axes\_base.pyc in _update_line_limits(self, line)
   1495         Figures out the data limit of the given line, updating self.dataLim.
   1496         """
-> 1497         path = line.get_path()
   1498         if path.vertices.size == 0:
   1499             return

C:\Anaconda\lib\site-packages\matplotlib\lines.pyc in get_path(self)
    863         """
    864         if self._invalidy or self._invalidx:
--> 865             self.recache()
    866         return self._path
    867 

C:\Anaconda\lib\site-packages\matplotlib\lines.pyc in recache(self, always)
    564                 x = ma.asarray(xconv, np.float_)
    565             else:
--> 566                 x = np.asarray(xconv, np.float_)
    567             x = x.ravel()
    568         else:

C:\Anaconda\lib\site-packages\numpy\core\numeric.pyc in asarray(a, dtype, order)
    460 
    461     """
--> 462     return array(a, dtype, copy=False, order=order)
    463 
    464 def asanyarray(a, dtype=None, order=None):

TypeError: float() argument must be a string or a number

Calling plot with a np.nan results in the point being silently ignored, which I believe should also happen for pd.NaT.

plot(np.nan,5)

Activity

  1. WeatherGod commented on Jan 14, 2015

    @WeatherGod
    Member

    how does pandas implement NaT? We don't have a dependency upon pandas, so
    we can't import it directly to see if they are the same.

    On Wed, Jan 14, 2015 at 11:27 AM, jvspl notifications@github.com wrote:

    Trying to plot the NaT (not a time) value of pandas gives an error.
    Feature request: treat NaT just like NaN from numpy.

    from pylab import *import pandas as pd
    plot(pd.NaT,5)


    TypeError Traceback (most recent call last)
    in ()
    ----> 1 plot(pd.NaT,5)

    C:\Anaconda\lib\site-packages\matplotlib\pyplot.pyc in plot(_args, *_kwargs)
    3091 ax.hold(hold)
    3092 try:
    -> 3093 ret = ax.plot(_args, *_kwargs)
    3094 draw_if_interactive()
    3095 finally:

    C:\Anaconda\lib\site-packages\matplotlib\axes_axes.pyc in plot(self, _args, *_kwargs)
    1372
    1373 for line in self._get_lines(_args, *_kwargs):
    -> 1374 self.add_line(line)
    1375 lines.append(line)
    1376

    C:\Anaconda\lib\site-packages\matplotlib\axes_base.pyc in add_line(self, line)
    1484 line.set_clip_path(self.patch)
    1485
    -> 1486 self._update_line_limits(line)
    1487 if not line.get_label():
    1488 line.set_label('_line%d' % len(self.lines))

    C:\Anaconda\lib\site-packages\matplotlib\axes_base.pyc in _update_line_limits(self, line)
    1495 Figures out the data limit of the given line, updating self.dataLim.
    1496 """
    -> 1497 path = line.get_path()
    1498 if path.vertices.size == 0:
    1499 return

    C:\Anaconda\lib\site-packages\matplotlib\lines.pyc in get_path(self)
    863 """
    864 if self._invalidy or self._invalidx:
    --> 865 self.recache()
    866 return self._path
    867

    C:\Anaconda\lib\site-packages\matplotlib\lines.pyc in recache(self, always)
    564 x = ma.asarray(xconv, np.float_)
    565 else:
    --> 566 x = np.asarray(xconv, np.float_)
    567 x = x.ravel()
    568 else:

    C:\Anaconda\lib\site-packages\numpy\core\numeric.pyc in asarray(a, dtype, order)
    460
    461 """
    --> 462 return array(a, dtype, copy=False, order=order)
    463
    464 def asanyarray(a, dtype=None, order=None):

    TypeError: float() argument must be a string or a number

    Calling plot with a np.nan results in the point being silently ignored,
    which I believe should also happen for pd.NaT.

    plot(np.nan,5)

    —
    Reply to this email directly or view it on GitHub
    #4000.

  2. added this to the milestone on Jan 14, 2015
  3. jvspl commented on Jan 14, 2015

    @jvspl
    Author

    How pandas implements NaT is beyond my knowledge. I cross-posted this issue on the pandas tracker as pandas-dev/pandas#9253

  4. tacaswell commented on Jan 14, 2015

    @tacaswell
    Member

    internally we convert input to float (because that makes sense for a plotting library) and the code is blowing up because float(pd.NaT) blows up.

    Surprisingly int(pd.NaT) does work, but return a very negative number which is probably more surprising than the code blowing up, so adding a try...except block to fall back to converting to int would be worse.

    The ideal thing would be for float(pd.NaT) to return np.nan, but I am not sure if that is possible.

    pandas imports matplotlib so we can not import pandas. The real solution to a lot of these problems is that all of the plotting should be pulled out of pandas and then either pandas becomes one of our dependencies or there is a third project which is a pandas-aware extension of mpl.

    In the mean time, I would suggest using pandas to either drop your NaT rows or replace them with np.nan.

    @jreback who should this get this pointed at on your side?

  5. jreback commented on Jan 15, 2015

    @jreback

    pd.NaT is a sub-class of datetime.datetime, that is treated like a missing value (it has similar properties as np.datetime64('NaT'). It is not-a-time. It is implemented similarly to np.datetime64('NaT') in that it is defined as np.iinfo(np.int64).min. This was a work-around a long time ago by @wesm to add missing value support to datetimes.

    It does not make sense to treat it as anything other than a datetime-like (which are represented by int64's under the hood).

    You can simply treat it as a datetime.datetime

    In [10]: pd.NaT
    Out[10]: NaT
    
    In [11]: pd.NaT.value
    Out[11]: -9223372036854775808
    
    In [12]: pd.NaT.year 
    Out[12]: nan
    
    In [13]: pd.NaT.month
    Out[13]: nan
    
    In [14]: type(pd.NaT.month)
    Out[14]: float
    
    In [15]: pd.NaT.minute
    Out[15]: nan
    
  6. shoyer commented on Jan 15, 2015

    @shoyer

    @tacaswell It's not super-clean, but it definitely is an option to make pandas an optional dependency of matplotlib, with import pandas never found at the top level of any modules but rather at the top of functions. In fact, this is exactly what pandas does (in reverse) with matplotlib.

  7. efiring commented on Jan 15, 2015

    @efiring
    Member

    @shoyer, I don't understand why we would want to do that. In which matplotlib functions would you bury this dependency?
    I wonder whether the problem of pandas inputs could be handled via our units support; could pandas classes be optionally added to the units.registry, just as datetime.date and datetime.datetime are?

  8. shoyer commented on Jan 15, 2015

    @shoyer

    @efiring I did not mean to suggest that that was a good idea, merely that it was possible! It does look like the matplotlib units library would be reasonable way to handle this -- probably worth a try.

  9. pelson commented on Feb 3, 2015

    @pelson
    Member

    Does the NaT not just need registering at https://github.com/pydata/pandas/blob/722fc778654f36092eccf9c0c654b03e97617aeb/pandas/tseries/converter.py#L27?

    For the record (and because I guess @shoyer will be interested) - I recently delved into the units code for the first time (in depth at least) - it is a pretty powerful (if perhaps over-engineered) piece of code, and I used it to implement support for netcdftime phony datetime objects so that I could plot 360-day calendars. Work in progress at https://gist.github.com/pelson/786ce4827aa1cd40b3f6 - eventually this will find its way to matplotlib/netcdftime/Iris/somewhere once I've rounded off the corners.

  10. shoyer commented on Feb 4, 2015

    @shoyer

    @pelson In theory, yes, I think that would do the trick. But I spent 15 minutes on this and it didn't work -- not quite sure why. See shoyer/pandas@62895af if you're curious or if anyone else is interested in following up on this.

    Also, glad to hear you're making progress on plotting netcdftime.datetime objects!

  11. petehuang commented on Jan 6, 2017

    @petehuang
    Contributor

    Hi,

    This hasn't been updated in a while but I am confirming in 1.5.3 that this still is not supported. It sounds like there's some work put in here though.

    Recommend removing upstream fix required as it doesn't sound like the burden is on pandas for the fix

  12. efiring commented on Jan 6, 2017

    @efiring
    Member

    @petehuang, my understanding of this issue is that it really is an upstream bug. When pandas is imported, it registers its own unit converter with matplotlib, and that converter is responsible for turning pd.NaT into a NaN. It's out of our hands. The converter does not seem to be handling either pd.NaT (which is a datetime.datetime) or np.datetime64('NaT') correctly. (Pandas registers a converter for np.datetime64, which matplotlib lacks, in addition to one for datetime.datetime.)

    The pd.NaT is a very strange beast. It is indeed a subclass of datetime.datetime, but instead of ints for its central data elements like the year, it uses floats, which allows those elements to be NaN.

  13. petehuang commented on Jan 6, 2017

    @petehuang
    Contributor

    Perfect, thank you for the explanation!

  14. anntzer commented on Jun 21, 2017

    @anntzer
    Contributor

    Closing, as the general agreement seems to be that this should be fixed on pandas' side. pandas devs should feel free to request a reopen if this changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions