Now I look at this more carefully, I think I like the original version using nested try blocks better. Reading the new code I have to correlate the two separate removals to prove to myself that as soon as we've awaited fut (whether it got a return value or was cancelled) it is removed from the list of waiters. And that's why it was written like that originally.

(Note that try blocks cause no overhead unless an exception is actually raised -- the compiler even duplicates the finally block to ensure this.)

Thanks, I guess it makes it simpler to reason about, good to know that nested blocks have no overhead anymore.

This comment makes things more mysterious to me, not less. If we're precise, we get to this exception handles whenever await fut raised something -- that can currently only be a cancellation, unless something reached into private data, extracted the future, and called set_exception() on it. (Apparently this is what you're planning to do in your own library?)

If I had to explain in plain English what was going on here, I'd have to say something like "The acquiring task was cancelled after the lock was released, but before it resumed execution. (This is one of these things that's easy to forget is possible when casually reading async code.) We're going to re-raise the cancellation here, but we should first wake up the next task waiting for the lock, if there are any." I guess you summarized the condition as "after release() was called", which I think is too concise.

Thinking all this over, I am also thinking there's an incredibly subtle invariant going on here regarding self._waiters. Skimming the code of release() and _wake_up_first() one starts wondering, "what if two or more waiters are cancelled, the first waiter wakes up (being cancelled), and its _wake_up_first() sees that the second waiter is also cancelled (and hence done), so it doesn't wake up the third waiter." My guess is that in this case the second waiter will also wake up and call _wake_up_first(), at which point the (originally) third waiter will be woken up. But maybe this deserves a reassuring comment somewhere?

Maybe there's a way of thinking about this that makes it easier to prove to oneself that the code here is correct, without such a "global" analysis? Maybe the invariant is something like "every cancelled future in the list corresponds to a task that will eventually be woken up, at which point it will remove itself from the list and call _wake_up_first(). Although that immediately makes me wonder if it is possible for the second waiter to be woken up first. Fortunately in that case things also work out, once the first waiter is also woken up.

Thank you for your thoughts. Yes, the comment was too concise and your version is better, but thinking more about it, it is not really true either. This here is not a case of "cancelled after lock was released". this is because release() does not do any modification of the _waiters queue itself. All it does is

1. set self._locked=False
2. Ensure that the Task owning the future at the head of the queue will wake up.

I go over the self._waiters in a comment below. Here is how I follow the logic:

1. _wake_up_first() always wakes up the same Task, at the head of the queue.
2. Only when a task resumes execution, does it remove itself from the queue, opening the possbility for a different Task to run. When it resumes execution, it either claims the lock for itself, or gives the task at the new head of the queue the chance to claim it.
3. When a task isn't about to claim the lock, and self._locked() is False, it is always safe to call _wake_up_first(), even multiple times.

Looking at the case you mention:

1. two waiters are cancelled. They still are in the self._waiters queue.
2. first one wakes up. Removes itself from self._waiters. Decides it doesn't want the lock, sees that self._locked is False (no one else has it), and so calls `self._wake_up_first(). This is a no-op, since the future has already been cancelled.
3. Second one wakes up. Removes itself from self._waiters. Decides it doesn't want the lock, seelf that self._locked is False, calls self._wake_up_first.
4. If there was a Task at the head, then it is now awoken.

Not sure how to formulate that into an invariant, but something like this:

if not self._locked and not claiming_lock, and self._waiters:
  # ensure that the head of the queue will run
  head = self.waiters[0]
  head.set_result(True)

In other words, if the lock is not claimed, then the head of the queue, if any, should no longer be blocked.

I'm still not very comfortable with widening this exception net for the purpose of sneaking in support for your library. If we want that to be supported we should make it an explicit feature, everywhere, rather than just tweaking an except clause here and there until your tests pass. Without any comments explaining the intended guarantee, what's to stop the next clever maintainer from narrowing the exception being caught to CancelledError again?

Fair point. I quite understand that you are reluctant to touch this just to humour an eccentric experimental library which is trying to push the envelope of the original design. And I'm happy to have this rejected as long as we have had the chance to discuss it. I'll add a comment here in the mean time, just for safety.

IIUC you're planning to make your library to use a subclass of CancelledError, so you don't need this to be more general anyway. So I'd like to see CancelledError here.

Sure. Of course, IMO, it would be even greater if in some future version, we would have something like:

class InterruptError(BaseException):
    pass

class CancelledError(InterruptError):
   pass

but that is future music :)

Same thing as before -- I actually like the nested try/finally phrasing better. Also, again, sneaky about catching all errors.

Fair point, especially if it costs nothing.

IIUC you're planning to make your library to use a subclass of CancelledError, so you don't need this to be more general anyway. So I'd like to see CancelledError here.

Nice comment! It doesn't immediately alleviate my fear that the future now at the head is already cancelled, and nothing will be woken up. The explanation is, of course, that all cancelled tasks in the waiters list are already woken up and are just waiting to run. Maybe you could add a hint about that?

(And maybe reflow the text, pretty please? In comments, you have to do the rendering in your head. :-)

I'd remove that -- I don't think there's such a thing as fut.true() and the semantics of set_result() ensure that if the future was cancelled it will raise an exception.

(Ditto if it already had another result, or an exception.)

Right, I added that as an explanation to what the state ought to be, to mirror the other test we make, but it is really redundant (and incorrect).

(Ditto if it already had another result, or an exception.)

Please convert the docstrings on the tests to comments -- test docstrings tend to be printed by the test runner, messing up the neat output. Note how no other tests here have docstrings.