Update typing to CPython 3.8 by fanninpm · Pull Request #2633 · RustPython/RustPython

fanninpm · 2021-05-12T00:09:39Z

(Eventually, if this ends up getting merged, this will close #2622 and close #2658.)

I haven't gotten this module to import cleanly if I mostly take it from CPython 3.8 sources:

>>>>> import typing
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/Documents/GitHub/RustPython/vm/pylib-crate/Lib/typing.py", line 1572, in <module>
    class SupportsAbs(Protocol[T_co]):
TypeError: '_ProtocolMeta' object is not subscriptable

Line 1572 is the first occurence of a class inheriting from Protocol[T_co] rather than Protocol. Did the syntax change between CPython 3.6 and CPython 3.7?

Let me know how we can get around this.

coolreader18 · 2021-05-12T01:01:47Z

Yeah, I think this involves implementing __class_getitem__ and GenericAlias. e.g. in CPython 3.9:

>>> list[int]
list[int]
>>> type(_)
<class 'types.GenericAlias'>

fanninpm · 2021-05-12T02:21:11Z

Yeah, I think this involves implementing __class_getitem__ and GenericAlias.

I don't think that first one is in the "what's left" file.

e.g. in CPython 3.9:

>>> list[int]
list[int]
>>> type(_)
<class 'types.GenericAlias'>

Compare CPython 3.8:

Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 07:01:05) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> getitem_args = []
>>> class C:
...     def __class_getitem__(*args, **kwargs):
...         getitem_args.extend([args, kwargs])
...         return None
... 
>>> C[int, str]
>>> getitem_args
[(<class '__main__.C'>, (<class 'int'>, <class 'str'>)), {}]

fanninpm · 2021-05-12T16:31:58Z

-class SupportsAbs(_Protocol[T_co]):
-    __slots__ = ()
-
-    @abstractmethod
-    def __abs__(self) -> T_co:
-        pass


@coolreader18 this didn't fail in CPython 3.6 and it doesn't fail in RustPython. The only difference here (compared to line 1572 in the new file) in the class definition is the lack of an underscore before Protocol[T_co].

fanninpm · 2021-05-16T23:04:53Z

I grep'd the 3.8 branch of the CPython source code for certain things, and here's what I found:

grep -Rn "__class_getitem__"

Modules/_testcapimodule.c:5895:    {"__class_getitem__", generic_class_getitem, METH_O|METH_CLASS, NULL},
Misc/NEWS.d/3.7.0a4.rst:19:``__class_getitem__`` is now an automatic class method.
Misc/NEWS.d/3.7.0a4.rst:175::pep:`560`: Add support for ``__mro_entries__`` and ``__class_getitem__``. Implemented
Misc/NEWS.d/3.8.1rc1.rst:190:Return class from ``ContextVar.__class_getitem__`` to simplify subclassing.
Misc/NEWS.d/3.8.0a2.rst:60:Fix ``__class_getitem__()`` not being called on a class with a custom
Doc/whatsnew/3.7.rst:411:The PEP introduces two special methods :meth:`__class_getitem__` and
Doc/library/typing.rst:234:The :class:`Generic` base class defines :meth:`__class_getitem__` so that
Doc/reference/datamodel.rst:2145:.. classmethod:: object.__class_getitem__(cls, key)
Lib/test/test_genericclass.py:154:            def __class_getitem__(*args, **kwargs):
Lib/test/test_genericclass.py:163:            def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:170:            def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:178:            def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:181:            def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:189:            def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:198:                def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:200:                cls.__class_getitem__ = classmethod(__class_getitem__)
Lib/test/test_genericclass.py:209:            def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:219:            def __class_getitem__(cls):
Lib/test/test_genericclass.py:224:            def __class_getitem__(cls, one, two):
Lib/test/test_genericclass.py:231:            def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:237:        e.__class_getitem__ = lambda cls, item: 'This will not work'
Lib/test/test_genericclass.py:241:            __class_getitem__ = "Surprise!"
Lib/test/test_genericclass.py:247:            def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:254:            def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:263:            def __class_getitem__(cls, item):
Lib/test/test_genericclass.py:264:                return 'from __class_getitem__'
Lib/test/test_genericclass.py:273:        self.assertIsInstance(Generic.__class_getitem__(int), GenericAlias)
Lib/typing.py:879:    def __class_getitem__(cls, params):
Lib/pydoc_data/topics.py:9360:                 'classmethod object.__class_getitem__(cls, key)\n'
Python/context.c:1028:    {"__class_getitem__", contextvar_cls_getitem,
Objects/abstract.c:176:        _Py_IDENTIFIER(__class_getitem__);
Objects/abstract.c:177:        if (_PyObject_LookupAttrId(o, &PyId___class_getitem__, &meth) < 0) {
Objects/typeobject.c:59:_Py_IDENTIFIER(__class_getitem__);
Objects/typeobject.c:2695:    /* Special-case __init_subclass__ and __class_getitem__:
Objects/typeobject.c:2712:    tmp = _PyDict_GetItemIdWithError(dict, &PyId___class_getitem__);
Objects/typeobject.c:2717:        if (_PyDict_SetItemId(dict, &PyId___class_getitem__, tmp) < 0) {

grep -Rn "GenericAlias"

Modules/_testcapimodule.c:5842:} PyGenericAliasObject;
Modules/_testcapimodule.c:5845:generic_alias_dealloc(PyGenericAliasObject *self)
Modules/_testcapimodule.c:5852:generic_alias_mro_entries(PyGenericAliasObject *self, PyObject *bases)
Modules/_testcapimodule.c:5862:static PyTypeObject GenericAlias_Type = {
Modules/_testcapimodule.c:5864:    "GenericAlias",
Modules/_testcapimodule.c:5865:    sizeof(PyGenericAliasObject),
Modules/_testcapimodule.c:5875:    PyGenericAliasObject *o = PyObject_New(PyGenericAliasObject, &GenericAlias_Type);
Modules/_testcapimodule.c:6353:    if (PyType_Ready(&GenericAlias_Type) < 0)
Modules/_testcapimodule.c:6355:    Py_INCREF(&GenericAlias_Type);
Modules/_testcapimodule.c:6356:    PyModule_AddObject(m, "GenericAlias", (PyObject *)&GenericAlias_Type);
Lib/test/test_genericclass.py:272:        from _testcapi import Generic, GenericAlias
Lib/test/test_genericclass.py:273:        self.assertIsInstance(Generic.__class_getitem__(int), GenericAlias)
Lib/test/test_genericclass.py:276:        self.assertIs(type(IntGeneric), GenericAlias)
Lib/typing.py:9:* The core of internal generics API: _GenericAlias and _VariadicGenericAlias, the latter is
Lib/typing.py:140:    if (isinstance(arg, _GenericAlias) and
Lib/typing.py:182:        if isinstance(t, _GenericAlias) and not t._special:
Lib/typing.py:191:    if not isinstance(tp, _GenericAlias):
Lib/typing.py:226:        if isinstance(p, _GenericAlias) and p.__origin__ is Union:
Lib/typing.py:271:    if isinstance(t, _GenericAlias):
Lib/typing.py:351:            return _GenericAlias(self, (item,))
Lib/typing.py:362:            return _GenericAlias(self, parameters)
Lib/typing.py:369:            return _GenericAlias(self, parameters)
Lib/typing.py:650:class _GenericAlias(_Final, _root=True):
Lib/typing.py:691:        return _GenericAlias(self.__origin__, params, name=self._name, inst=self._inst)
Lib/typing.py:712:        if not isinstance(other, _GenericAlias):
Lib/typing.py:742:            if not any(isinstance(b, _GenericAlias) or issubclass(b, Generic)
Lib/typing.py:751:                if isinstance(b, _GenericAlias) and b is not self:
Lib/typing.py:773:            if not isinstance(cls, _GenericAlias):
Lib/typing.py:798:class _VariadicGenericAlias(_GenericAlias, _root=True):
Lib/typing.py:799:    """Same as _GenericAlias above but for variadic aliases. Currently,
Lib/typing.py:898:        return _GenericAlias(cls, params)
Lib/typing.py:918:                if (isinstance(base, _GenericAlias) and
Lib/typing.py:1285:    if isinstance(tp, _GenericAlias):
Lib/typing.py:1303:    if isinstance(tp, _GenericAlias) and not tp._special:
Lib/typing.py:1436:    return _GenericAlias(origin, params, special=True, inst=inst)
Lib/typing.py:1449:Callable = _VariadicGenericAlias(collections.abc.Callable, (), special=True)
Lib/typing.py:1468:Tuple = _VariadicGenericAlias(tuple, (), inst=False, special=True)
Lib/dataclasses.py:599:            or (type(a_type) is typing._GenericAlias

grep -Rn "not subscriptable"

Lib/test/test_grammar.py:1355:        msg=r'is not subscriptable; perhaps you missed a comma\?'
Lib/typing.py:370:        raise TypeError(f"{self} is not subscriptable")
Python/compile.c:3959:        return compiler_warn(c, "'%.200s' object is not subscriptable; "
Objects/abstract.c:187:    return type_error("'%.200s' object is not subscriptable", o);

Of the actual code files, I guess we can safely ignore Modules/_testcapimodule.c and Lib/test/test_genericclass.py for now, and I know that Objects/typeobject.c roughly corresponds to vm/src/builtins/pytype.rs. ~~However, what does Objects/abstract.c correspond to?~~

EDIT: I found vm/src/pyobject.rs, but where can I insert something that has the same function as the following CPython lines (in Objects/abstract.c)?

https://github.com/python/cpython/blob/4844abdd700120120fc76c29d911bcb547700baf/Objects/abstract.c#L174-L185

fanninpm · 2021-05-18T14:46:53Z

+    #[pyclassmethod(magic)]
+    fn class_getitem(cls: PyTypeRef, _args: FuncArgs) -> PyObjectRef {
+        cls.into_object()
+    }


@coolreader18 Now that this is in the codebase, the next step is to tackle this problem TypeError: @runtime_checkable can be only applied to protocol classes, got <class 'typing.SupportsAbs'>. I don't think a class is inheriting correctly.

If I remove the decorators I can see that the protocols are inheriting from typing._ProtocolMeta, which is the result of typing.Protocol[typing.T_co]. In CPython typing.Protocol[typing.T_co] returns an instance of typing._GenericAlias.

If I make my own class with __class_getitem__ the method is never called and type is always returned. If the class has a different metaclass that's returned instead.

I added some prints to the get_item method in pyobject.rs, and from that I can see vm.get_special_method(obj, "__class_getitem__") returns the method for type, rather than for the class itself. Again, if there's a metaclass its the method for the metaclass. Shouldn't this be calling the method on the class?

Note on these changes. Why not add them in a separate PR? They will eventually be added, might as well add them before typing (since typing might need more time to land, right?).

From my perspective, it makes more sense to reduce the number of PRs that have to be merged before this one, i.e. the dependencies of this PR. For really large changes it's good to split out into distinct features, but for relatively small ones it's fine to just have them as commits.

fanninpm · 2021-05-18T18:15:02Z

@oroppas you might want to follow the progress on this.

coolreader18 · 2021-05-19T05:35:19Z

@fanninpm do you have an auto formatter in your editor? It looks like formatting changed in the most recent commit

fanninpm · 2021-05-19T21:24:49Z

@fanninpm do you have an auto formatter in your editor? It looks like formatting changed in the most recent commit

@coolreader18 Thanks for catching that! I forgot to turn off editor.formatOnSave. Should be fixed in 8ba36ab.

DimitrisJim · 2021-06-11T19:06:28Z

That CI error seems to be because classes defined in Python are inheriting descriptors from type. __qualname__ isn't allowed to be set for builtins afaik but for Python classes you are allowed to set them to whatever you want.

DimitrisJim · 2021-08-18T03:14:41Z

@fanninpm if you can, make your most recent push to ignore test_acquire_contended a separate PR. This issue was introduced in #2902 and is stalling the rest of the pull requests so I'd be good to get it merged as soon as possible.

Update: #2905

c.f. lines 2695-2725 of typeobject.c in CPython source

@coolreader18

This restores an alteration made by @coolreader18 a while back.

They will be fixed later.

youknowone · 2021-08-21T14:21:19Z

@fanninpm @coolreader18 is there further works need to be done here?

coolreader18

LGTM! Surprisingly small amount of changes to get this working :)

coolreader18 · 2021-08-21T19:57:48Z

Although idk, maybe GenericAlias should be implemented as well? I don't think typing is really tested that much in CI.

coolreader18 · 2021-08-21T20:00:41Z

Meh, it'll hopefully be not too hard to implement.

coolreader18 · 2021-08-21T20:12:38Z

Thanks for working on this @fanninpm !

coolreader18 · 2021-08-21T21:11:33Z

Ah, actually, 3.8 has no built-in GenericAlias.

fanninpm commented May 12, 2021

View reviewed changes

fanninpm force-pushed the update-typing branch from f68aa55 to 3322d1e Compare May 18, 2021 02:31

fanninpm commented May 18, 2021

View reviewed changes

fanninpm force-pushed the update-typing branch from 3fe2af9 to 8ba36ab Compare May 19, 2021 21:23

fanninpm force-pushed the update-typing branch from 8ba36ab to 0f6bb11 Compare May 23, 2021 16:30

fanninpm force-pushed the update-typing branch from 497c172 to 0a233ae Compare July 11, 2021 18:34

fanninpm force-pushed the update-typing branch from 0a233ae to 20b9eb1 Compare August 3, 2021 00:12

fanninpm force-pushed the update-typing branch from 20b9eb1 to 701b631 Compare August 18, 2021 00:32

fanninpm added 6 commits August 18, 2021 20:08

Update typing to CPython 3.8

fab274a

Make __class_getitem__ always a class method

3921ea4

c.f. lines 2695-2725 of typeobject.c in CPython source

Implement "__class_getitem__" (barely working)

48de035

Add '_py_abc' to the frame stack

79779f8

This restores an alteration made by @coolreader18 a while back.

Comment out sections of typing.py that don't work

0892e54

They will be fixed later.

Remove namespace conflicts with io and re

650b60c

fanninpm force-pushed the update-typing branch from 2d75f79 to 650b60c Compare August 19, 2021 00:13

coolreader18 mentioned this pull request Aug 21, 2021

Bump typing module to 3.8 #2918

Closed

moreal mentioned this pull request Aug 21, 2021

Support generic class (i.e. __class_getitem__) #2917

Closed

Fix pip with new typing

629fadf

fanninpm marked this pull request as ready for review August 21, 2021 16:38

coolreader18 approved these changes Aug 21, 2021

View reviewed changes

coolreader18 merged commit 4f0feef into RustPython:master Aug 21, 2021

fanninpm deleted the update-typing branch August 21, 2021 22:27

moreal mentioned this pull request Aug 22, 2021

Missing dataclasses module #2623

Closed

Conversation

fanninpm commented May 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coolreader18 commented May 12, 2021

Uh oh!

fanninpm commented May 12, 2021

Uh oh!

fanninpm May 12, 2021

Choose a reason for hiding this comment

Uh oh!

fanninpm commented May 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fanninpm May 18, 2021

Choose a reason for hiding this comment

Uh oh!

domdfcoding May 18, 2021

Choose a reason for hiding this comment

Uh oh!

DimitrisJim Jun 7, 2021

Choose a reason for hiding this comment

Uh oh!

coolreader18 Jun 8, 2021

Choose a reason for hiding this comment

Uh oh!

fanninpm commented May 18, 2021

Uh oh!

coolreader18 commented May 19, 2021

Uh oh!

fanninpm commented May 19, 2021

Uh oh!

DimitrisJim commented Jun 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DimitrisJim commented Aug 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

youknowone commented Aug 21, 2021

Uh oh!

coolreader18 left a comment

Choose a reason for hiding this comment

Uh oh!

coolreader18 commented Aug 21, 2021

Uh oh!

coolreader18 commented Aug 21, 2021

Uh oh!

coolreader18 commented Aug 21, 2021

Uh oh!

coolreader18 commented Aug 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fanninpm commented May 12, 2021 •

edited

Loading

fanninpm commented May 16, 2021 •

edited

Loading

DimitrisJim commented Jun 11, 2021 •

edited

Loading

DimitrisJim commented Aug 18, 2021 •

edited

Loading