Skip to content

Conversation

@pablogsal
Copy link
Member

@pablogsal pablogsal commented Aug 20, 2024

Python/ast_opt.c Outdated
Comment on lines 610 to 638
if (newval == NULL) {
return 0;
}
Copy link
Member

@picnixz picnixz Aug 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fold_tuple, this call is immediately followed by make_const which then clears any exception that is not PyExc_KeyboardInterrupt, and then returns 1. So my question is: should we actually clear the error or not in this case as well? (or maybe patch fold_tuple in this case?).

More generally, calls to make_const are not preceeded by newval == NULL checks.

Copy link
Member Author

@pablogsal pablogsal Aug 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that make_const does error checking for one of their arguments and callers are relying on that behaviour is not only error prone but it's also very confusing. If we fail to create one of this objects we should propagate the error (think about MemoryError or something worse, we surely don't want to clean that).

In other words, calls to make_const are not preceeded by newval == NULL checks.

I think that's fundamentally flawed. This code was moved from the time this happened in peephole.c (and even before, if you track it down it goes as back as 00ee7ba) and was clearing out all exceptions in the peephole after compilation and special casing KeyboardInterrupt which it's masking real errors and can actually crash if the caller is left in an inconsistent state.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should clean this up in another PR

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I just wanted to address the test failures (which I think they could be related to this). But I agree that we shouldn't clear errors otherwise. I was more concerned about: "should we fix calls to make_const here or separately?".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that make_const does error checking for one of their arguments and callers are relying on that behaviour is not only error prone but it's also very confusing. If we fail to create one of this objects we should propagate the error (think about MemoryError or something worse, we surely don't want to clean that).

In other words, calls to make_const are not preceeded by newval == NULL checks.

I think that's fundamentally flawed. This code was moved from the time this happened in peephole.c (and even before, if you track it down it goes as back as 00ee7ba) and was clearing out all exceptions in the peephole after compilation and special casing KeyboardInterrupt which it's masking real errors and can actually crash if the caller is left in an inconsistent state.

Let's think about one concrete case: 1 / 0. There's ZeroDivisionError is being raised.
Should we propagate it? That's a really a complex question.
If you don't mind, I would like to work on this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should always propagate. If there is an exception we screwed up somewhere and the code should defend against it otherwise. Reliying on a catch-all it's bad because this is basically ignoring anything inheriting from BaseException, including SystemErrors

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here the other problem really is that make_const_tuple it's returning NULL without the error set :(

Copy link
Member Author

@pablogsal pablogsal Aug 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have made here a big refactor so make_const doesn't eat errors and so exceptions that are explicitly ignored are the only thing that's ignored

@pablogsal
Copy link
Member Author

converting to draft meanwhile I try to address test failures

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
@serhiy-storchaka
Copy link
Member

Is this the only way to fix an error (whatever it is)? Currently the complexity of errors handling is hidden in make_const(), and the rest of code is kept clear. This PR spreads complexity across the whole file. I am not a fun of such change.

What exactly issues with the current code? I will try to solve them in other way.

@pablogsal
Copy link
Member Author

pablogsal commented Aug 20, 2024

Currently the complexity of errors handling is hidden in make_const()

The first issue is that that's very confusing because if you don't know that make_const it's cleaning the error indicator, it looks like a leak and that would be an anti-pattern everywhere else in the code. Furthermore, if someone copies that function it will quickly introduce actual leaks if there is an error path after because the argument will not be explicitly captured. For instance if you do

int res = make_const(node_, PyBool_FromLong(!state->optimize), ctx_); // Code copied from other call
if (!res) {
    return 0;
}
PyObject* my_list = Py_ListNew(0);
if (!my_list) {
    return 0; // Whops! Forgoto to clean the bool
}

A reviewer that doesn't know this weird behaviour about make_const() will be very confused (I was in #123081 (comment))

The second thing it's that is capturing everything and clearing it (except KeyBoardInterrupt) which isn't granular at all and it will hide other things like MemoryError and other exceptions that may happen there for any reason.

This PR spreads complexity across the whole file

It just surfaces existing complexity and makes the error paths more obvious. It's like if you say that doing

try:
   ...
except KeyError:
   ...
except ZeroDivisioError
   ...
else:
   print("BAD")
   raise

instead of

try:
   .... a lot of code here ...
except:
   pass

it's adding a lot of complexity. Sure there is more code, but you are being explicit about what's being ignored and why (like the ZeroDivisionError) and it's not hiding errors that may be legitimate .

@serhiy-storchaka OTOH I don't feel very strongly about this so If you still think you prefer the current code anyway I'm happy to close this but

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented Aug 21, 2024

This is not an antipattern. If there is some code repeated multiple times, you can factor out it into a function. And if this is the same check of argument before every call of the function, it can be moved into that function. This simplifies multiple caller sites and guarantees that the check will be not missed.

Such pattern is not common in the public C API, this is why it can be confusing, but it is used for convenience if this is appropriate (see for example Py_BuildValue("N") and PyModule_Add()). It is more often used in local functions.

Usually it is more correct to check for exceptions types that should be silenced (OverflowError, MemoryError, RecursionError, whatever) instead of exceptions types that should not be silenced (KeyboardInterrupt), but what other exceptions can be raised here? There is a limited set of exceptions that can be raised here, and only KeyboardInterrupt should not be silenced. If we a wrong and missed some type of very rare exception that should be silenced, we do not want to throw it in the face of a user. I do not know any other exception besides KeyboardInterrupt that should not be ignored. So checking for KeyboardInterrupt is not only practical (less code), it is more error-proof.

I prefer the current code.

@pablogsal
Copy link
Member Author

pablogsal commented Aug 21, 2024

This is not an antipattern. If there is some code repeated multiple times, you can factor out it into a function. And if this is the same check of argument before every call of the function, it can be moved into that function. This simplifies multiple caller sites and guarantees that the check will be not missed.

I don't see this as refactoring common code, the function is called make_const not maybe_make_const or something that suggest that it may receive null or clear errors. The fact that it checks the argument and handles error is not apparent from the name nor in any reviews.

What's more, we already have problems with functions that swallor errors and new functions had to be introduced to fix the confision for users (see PyDict_GetItemWithError).

but what other exceptions can be raised here?

At the very least any of the calls that create Python objects can raise MemoryError and technically if signal handlers are installed then the signal handler can raise whatever exception and we will clear it.

Such pattern is not common in the public C API, this is why it can be confusing, but it is used for convenience if this is appropriate (see for example Py_BuildValue("N") and PyModule_Add()). It is more often used in local functions.

I disagree, in the first case (Py_BuildValue) you can tell what would happen with lifetimes from the caller site because of the N and PyModule_Add doesn't clear any existing error that I know.

What also worries me more is that there may be new exceptions appearing after refactors that are going to be ignored and the optimisations may not be performed and we will not find out easily unless the tests are triggering those problems as well, which is unlikely.

@pablogsal
Copy link
Member Author

I'm closing this, I don't feel that passionate about it and I trust @serhiy-storchaka criteria

@pablogsal pablogsal closed this Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants