Repository navigation
Description
I tried out a couple of exercises on the Python track today, and noticed with the Raindrops exercise in particular that the tests were requiring me to implement the entire algorithm on the second failure.
I'm working in Python 3.8.5.
I run the tests with the following command:
pytest -x raindrops_test.py
With the unchanged stub file in place, I get the following failure:
def test_2_to_the_power_3_does_not_make_a_raindrop_sound_as_3_is_the_exponent_not_the_base(
self
):
> self.assertEqual(convert(8), "8")
E AssertionError: None != '8'
This is a reasonable "first" failure, since it's a single number that should be turned into a string.
If I make that pass with a hard-coded return "8", then this is the next failure I get:
def test_the_sound_for_105_is_pling_plang_plong_as_it_has_factors_3_5_and_7(self):
> self.assertEqual(convert(105), "PlingPlangPlong")
E AssertionError: '8' != 'PlingPlangPlong'
E - 8
E + PlingPlangPlong
Looking at the tests, the order that they are defined in is quite nice, in terms of pushing the implementation forward step by step.
I tried adding the following declaration after the import unittest statement, to see if this would cause the tests to be run in the order that they are defined:
unittest.TestLoader.sortTestMethodsUsing = None
This did not change the order that the tests are run in.
If I sort the tests alphabetically by name, it appears to correspond to the order that I'm seeing when doing the exercise:
While under normal circumstances I would want tests to be run in random order, in Exercism exercises I think that it would make more sense to run the tests in the order that they are defined.
Is there a declaration we can make in order to accomplish this?
If not, then I would propose that we rename tests by adding a numeric prefix to the test name (e.g.
def test_01_the_sound_for_1_is_1(self):
Activity
kytrinyx commentedon Aug 3, 2020
I did a search, and it looks like there has been some discussion about this previously:
iHiD commentedon Aug 3, 2020
@yawpitch said in #1853 (comment):
My quite take on this is that, while that's true and there are lots of places that the tests may not have been ordered correctly, there are also a significant amount of places where they have been deliberately ordered and where that order greatly improves the learning experience.
So there is value to respecting the order, although it will not always be beneficial, and I can't see the downside of not respecting the order.
If that is accurate, then the only negative of implementing some solution that respects the order is that (as outlined in #1853 and other places) there's not a particularly clean solution. However, I would argue that an ugly solution is better than not having ordering. So, pending someone willing to do the work, I think this is a change we should make.
yawpitch commentedon Aug 3, 2020
Piping in to explain: the technical reason we never did this is because
unittest.TestLoader.sortTestMethodsUsing = Nonedoes not imply line order sort, it merely removes entirely any current sort behavior (which is alpha by default)... so instead what you get is an undocumented and undefined order, though in practice in the current source code that’s the implicit sort order returned by callingdir(YourTestCaseSubclass)which, annoyingly, is alpha sort.To reliably get line order you need to set the
sortTestMethodsUsingclass attribute to a function that looks up the line number in the source code of the original file via runtime introspection and compares based on that... such a function is non-trivial, involves importing at least theinspectmodule, and would need to be added to the template for every generated test file. It’s also fairly brittle if inheritance is used to cobble together unittest.TestCase instances, though I can’t think of an example of that in our code base.All of that, and the fact that it just generally seems a bad idea to imply to students that they should expect a fully curated sort order in the real world of Python development, has weighed against this change.
Now, yes, another simpler way would be to add to the templates some test name generator logic that would introduce a sorting number (so each test method would be generated as “test_0001_some_aspect”, “test_0002_some_other_aspect”, etc), but we’d have to ensure zero padding for unknown numbers of tests, and we’ve already got a problem with overlong test names.
iHiD commentedon Aug 3, 2020
@yawpitch Thanks for explaining :)
In #1853 you mused about pytest saying:
Is there a (good enough) reason not to use pytest as the "blessed" runner? I believe many other tracks do have a specific test runner that they advocate (the JS family as an example). It sounds like this would simplify the situation and maybe give us a cleaner solution? But I don't know enough about Python to understand the tradeoffs.
cmccandless commentedon Aug 3, 2020
Reasons not to restrict to single test runner (I don't know if any of them are "good enough"):
pytest, currently the test code only requires a supported Python installation; no other dependencies required.pytestat all (I am not in favor of moving in this direction)pytestrunner, and just wants to practice with that, they are able to do so.AnAccountForReportingBugs commentedon Aug 20, 2020
These exercises are supposed to teach the basics, so I think it makes more sense to have just unittest usage explained to avoid the issue of people having to figure out how to install pytest. I don’t really see the benefit of pytest over unittest here, other than slightly prettier output. If someone is already familiar with another runner, I would think they’d know how to use it already regardless.
stale commentedon Sep 11, 2020
This issue has been automatically marked as
abandonedbecause it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.BethanyG commentedon Jul 14, 2021
I am going to re-open this issue. In light of V3 changes that now depend on
PyTestand the test runner interface for V3concept exerciseshaving a task number and implementation tests (runner interface specs), it no longer makes sense to argue that we are "only" usingunittest.We now have dependancies on
PyTest, andPyTestis at the heart of the runner that will be used on the V3 Website to run student code -- so we should instruct students to also install the same test runner and configuration on their own machines, so that they can use the same toolset prior to uploading.PyTestis very popular (I'd almost call it a default at this point) and is used for most Python projects (outside of the Python language core dev team which usesunittest) -- nose has been in maintenance mode for a while, and no one has stepped forward to continue development. nose2 is also largely in a holding pattern. There is ptr - but it is pretty new. The vast majority of the instruction or tutorials a student might encounter would point them toPyTest, orPyTest+ plugins. See this and this for some examples.Not going to go into ordering atm, but pytest-order and pytest-ordering as plugins to PyTest look interesting as options.