Skip to content

Meta issue: Mark all CPython-specific tests with a test.support.cpython_only decorator #115142

Open

Description

@Eclips4

Bug report

Bug description:

There's a certain amount of Cpython-specific tests which isn't marked with a cpython_only decorator.

I'm certain that CPython-specific tests should be excluded from running on any other implementation, which is advantageous for us. This ensures that when developing another Python implementation, you can selectively avoid running these specific CPython tests, which is beneficial during the testing phase. This way, you can focus on relevant tests for your implementation without unnecessary interference from CPython-specific ones.

So, my final goal is to make the whole CPython test suite runnable on other Python implementations.

I would like to send a one PR per test file which needs a fix.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

Activity

  1. added
    type-bugAn unexpected behavior, bug, or error
    testsTests in the Lib/test dir
    on Feb 7, 2024
  2. changed the title [-]Meta issue: Mark all cpython specific tests with a `test.support.cpython_only` decorator[/-] [+]Meta issue: Mark all CPython-specific tests with a `test.support.cpython_only` decorator[/+] on Feb 7, 2024
  3. serhiy-storchaka commented on Feb 7, 2024

    @serhiy-storchaka
    Member

    How are you going to determine that they are CPython-specific?

  4. Eclips4 commented on Feb 7, 2024

    @Eclips4
    MemberAuthor

    How are you going to determine that they are CPython-specific?

    It is a good question.
    First of all I would like to "transform" all tests which using CPython-specific modules (for example, _testinternalcapi and _elementtree)
    Secondly, if test using a some private (I mean non-documented) attributes, does this mean that this test is CPython specific?

  5. terryjreedy commented on Feb 7, 2024

    @terryjreedy
    Member

    This is a complicated issue that to me needs some meta-level discussion and possible inclusion of representatives of other implementations. I have several questions.

    1. Benefits to who? There are Python developers, CPython d Quoting the initial message

    I'm certain that CPython-specific tests should be excluded from running on any other implementation,

    Why? The answer depends on how broadly or narrowly 'CPython-specific' is defined.

    which is advantageous for us.

    'Us' as Python developers, or as CPython developers? I am responding mostly as a Python developer whose C knowledge has faded somewhat.

    This ensures that when developing another Python implementation, you can selectively avoid running these specific CPython tests, which is beneficial during the testing phase.

    cpython-only is all or nothing for the items marked. See next question.

    1. Skip level: assert, method, class, or module? I believe there was a recent patch that added perhaps a hundred decorators to a hundred methods. I would not like to read or edit such a file. If I were implementing the tested module, I doubt I would want to have to remove or comment out the hundred or so decorators one by one. I think most of the skipping should be at the class or even module level. If unittest cannot handle a class being skipped when called, as opposed to when it calls setUpModule, we should try to change it.

    Perhaps implementation skips of asserts should be discouraged. There may be some OS-dependent assert in IDLE test, but that is another matter.

    1. C-specific or CPython-specific? Which is true of C-coded modules like _elementtree? Is it not usable by any implementation that is either based on or can access C code? Things that are only C-specific should not be marked CPython-specific. In other words, do C-coded stdlib modules use the private C-API, however defined?

    2. Python objects: Public Python objects should not be CPython specific. Some Python-visible behaviors are and when possibly depended on is a manner we don't want, are documented as such. Still, if other implementations wish to imitate the behavior, they need to easily run the tests. Same comment for Python-coded private objects.

    If implementations that copy the module code also copy the test code, what is the problem? Or perhaps, should there be a problem? Do we have it as a goal that (with rare documented exceptions) Python-coded stdlib modules only depend on Python, and not CPython-specific behaviors? Should they avoid, for instance, depending on CPython's specific int caching? One way to test and catch this, would be to have an alternate test build that behaved otherwise, such as by disabling the caching.

  6. serhiy-storchaka commented on Feb 8, 2024

    @serhiy-storchaka
    Member

    See also #69317. PyPy uses modified CPython tests. We can look at that modifications and decide which of them are CPython-specific and which are PyPy-specific.

  7. Eclips4 commented on Feb 8, 2024

    @Eclips4
    MemberAuthor

    This is a complicated issue that to me needs some meta-level discussion and possible inclusion of representatives of other implementations. I have several questions.

    1. Benefits to who? There are Python developers, CPython d Quoting the initial message

    I'm certain that CPython-specific tests should be excluded from running on any other implementation,

    Why? The answer depends on how broadly or narrowly 'CPython-specific' is defined.

    which is advantageous for us.

    'Us' as Python developers, or as CPython developers? I am responding mostly as a Python developer whose C knowledge has faded somewhat.

    Sorry, I was unclear here. It's not make any benefit for CPython developers, but there are obvious benefits for Python ecosystem - testing other Python implementations will be easier.

    Skip level: assert, method, class, or module? I believe there was a recent patch that added perhaps a hundred decorators to a hundred methods. I would not like to read or edit such a file. If I were implementing the tested module, I doubt I would want to have to remove or comment out the hundred or so decorators one by one. I think most of the skipping should be at the class or even module level. If unittest cannot handle a class being skipped when called, as opposed to when it calls setUpModule, we should try to change it.

    That's depends on test which we'd like to transform. Tests like test_optimizer definitely should be skipped on module level.
    For instance, if some method testing CPython int caching, this method should be decorated with a cpython_only.

    C-specific or CPython-specific? Which is true of C-coded modules like _elementtree? Is it not usable by any implementation that is either based on or can access C code? Things that are only C-specific should not be marked CPython-specific. In other words, do C-coded stdlib modules use the private C-API, however defined?

    It's a good question! I think we talking about CPython-specific tests. For example, there are tests for _elementtree module (which is actually a C-extension). Do we need to skip tests for this module on non-CPython implementations? I think no. This module is not required to be present in stdlib, but what prevents other implementation from having it? So in my opinion we must skip this test if _elementtree module is not available.

  8. added a commit that references this issue on Feb 8, 2024
  9. Eclips4 commented on Feb 8, 2024

    @Eclips4
    MemberAuthor

    One more detail: I think these kind of changes should be backportable where it is possible.

  10. added a commit that references this issue on Feb 9, 2024
  11. added a commit that references this issue on Feb 14, 2024
  12. added a commit that references this issue on Mar 9, 2024
  13. added a commit that references this issue on Mar 9, 2024
  14. added a commit that references this issue on Mar 9, 2024
  15. added a commit that references this issue on Mar 25, 2024
  16. added 2 commits that reference this issue on Apr 5, 2024
  17. added a commit that references this issue on Apr 10, 2024
  18. added a commit that references this issue on Apr 10, 2024
  19. added 2 commits that reference this issue on Apr 11, 2024
  20. added 3 commits that reference this issue on Apr 17, 2024
  21. added a commit that references this issue on Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions