Skip to content

All exceptions are now modified with extend_exception! macro#2974

Merged
youknowone merged 14 commits into
RustPython:mainfrom
sobolevn:extends-exception-macro
Sep 15, 2021
Merged

All exceptions are now modified with extend_exception! macro#2974
youknowone merged 14 commits into
RustPython:mainfrom
sobolevn:extends-exception-macro

Conversation

@sobolevn

@sobolevn sobolevn commented Aug 26, 2021

Copy link
Copy Markdown
Contributor

So, I've decided to go with the simplest extend_exception! I can think of.

Why?

  1. It solves the problem
  2. It is flexible enough to write any code we want
  3. We already had something similar in pub fn extend(ctx: &PyContext)
  4. We don't actually need exception structs
  5. We cannot write fancy things with module level declarations, because we don't have ctx yet

Update: it was really hard to make it work with subclasses ad custom __new__ and __init__ in mind.
So, I've switched to a more complex struct-based macro.

Closes #2973
Refs #2960
Refs #2897

@sobolevn

Copy link
Copy Markdown
Contributor Author

@youknowone quick question: these lines https://github.com/RustPython/RustPython/pull/2974/files#diff-d19484f26ba252a3b3ff27867db9682f948572dd37e83e811d6f073bfc2f07c2R547-R548 are not technically correct. We don't need to always call PyBaseException::tp_new / init. We need to call supertype's new / init. What's the easiest way to do that?

@youknowone

youknowone commented Aug 27, 2021

Copy link
Copy Markdown
Member

Here is a rough idea:

  • object.__new__ must call the type's designated tp_new
  • other types __new__ call need to track up to object to call object.__new__
    Then they always correct tp_new will be called.

And other stuffs:
It will require additional mro traverse but will be compatible to CPython.
When I discussed about this, I've heard CPython has different slot management policy. So they copy the slot of base class when they create a new class. I think we finally need to follow the same way to remove mro tracking.

@sobolevn

Copy link
Copy Markdown
Contributor Author

@youknowone it is hard to understand what do you mean without a code-snippet 😞

@youknowone

Copy link
Copy Markdown
Member

I think the part is not directly related to this issue. You can keep it as current behaviour for now. We need to fix it for entier __new__, not just for exceptions.

@eldpswp99

Copy link
Copy Markdown
Contributor

Here is a rough idea:

  • object.__new__ must call the type's designated tp_new
  • other types __new__ call need to track up to object to call object.__new__
    Then they always correct tp_new will be called.

And other stuffs:
It will require additional mro traverse but will be compatible to CPython.
When I discussed about this, I've heard CPython has different slot management policy. So they copy the slot of base class when they create a new class. I think we finally need to follow the same way to remove mro tracking.

can i fix this issue?

@sobolevn

sobolevn commented Aug 28, 2021

Copy link
Copy Markdown
Contributor Author

Ok, new concept. I was not able to fix this problem with __new__ slot. So, I wrote a new macro that has __new__ as a parameter.

Now, this works:

define_exception! {
    PyPermissionError,
    PyOSError,
    permission_error,
    "Not enough permissions.",
    os_error_custom_new,
}

fn os_error_custom_new(
        cls: PyTypeRef,
        args: FuncArgs,
        vm: &VirtualMachine,
) -> PyResult<PyRef<PyBaseException>> {
        println!("custom new");
        PyBaseException::new(args.args, vm).into_ref_with_type(vm, cls)
}

But, for some reason I experience [1] 2494 segmentation fault RUST_BACKTRACE=1 cargo run locally. And cannot find out why 🙂

@sobolevn

Copy link
Copy Markdown
Contributor Author

I am trying to debug what is going on with valgrind. Not really helpful for now.

» cargo valgrind run

    Finished dev [unoptimized + debuginfo] target(s) in 0.58s
     Running `/Users/sobolev/.cargo/bin/cargo-valgrind target/debug/rustpython`
error: invalid valgrind usage: --20007-- run: /usr/bin/dsymutil "target/debug/rustpython"
--20007-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option

valgrind: m_signals.c:1106 (void handle_SCSS_change(Bool)): Assertion 'ksa_old.sa_flags == skss_old.skss_per_sig[sig].skss_flags' failed.

host stacktrace:
==20007==    at 0x25804212B: ???
==20007==    by 0x2580424A1: ???
==20007==    by 0x258042484: ???
==20007==    by 0x258052F7E: ???
==20007==    by 0x258052C2A: ???
==20007==    by 0x2580CFF19: ???
==20007==    by 0x2580BC6A1: ???
==20007==    by 0x2580BAFD4: ???
==20007==    by 0x2580B8E4E: ???
==20007==    by 0x2580CA28B: ???

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable syscall unix:46 (lwpid 771)
  <stack>
    <frame>
      <ip>0x10439FE26</ip>
      <obj>/usr/lib/system/libsystem_kernel.dylib</obj>
      <fn>__sigaction</fn>
    </frame>
    <frame>
      <ip>0x1043E352A</ip>
      <obj>/usr/lib/system/libsystem_platform.dylib</obj>
      <fn>__platform_sigaction</fn>
    </frame>
    <frame>
      <ip>0x1041BE6C3</ip>
      <obj>/usr/lib/system/libsystem_c.dylib</obj>
      <fn>signal__</fn>
    </frame>
    <frame>
      <ip>0x1003EE566</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::stdlib::signal::make_module</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src/stdlib</dir>
      <file>signal.rs</file>
      <line>277</line>
    </frame>
    <frame>
      <ip>0x100191C28</ip>
      <obj>target/debug/rustpython</obj>
      <fn>core::ops::function::Fn::call</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops</dir>
      <file>function.rs</file>
      <line>70</line>
    </frame>
    <frame>
      <ip>0x1003C5A03</ip>
      <obj>target/debug/rustpython</obj>
      <fn>&lt;alloc::boxed::Box&lt;F,A&gt; as core::ops::function::Fn&lt;Args&gt;&gt;::call</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/alloc/src</dir>
      <file>boxed.rs</file>
      <line>1560</line>
    </frame>
    <frame>
      <ip>0x100845DBD</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::import::import_builtin::{{closure}}</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src</dir>
      <file>import.rs</file>
      <line>103</line>
    </frame>
    <frame>
      <ip>0x100D6B3FB</ip>
      <obj>target/debug/rustpython</obj>
      <fn>core::result::Result&lt;T,E&gt;::and_then</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src</dir>
      <file>result.rs</file>
      <line>704</line>
    </frame>
    <frame>
      <ip>0x100845C71</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::import::import_builtin</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src</dir>
      <file>import.rs</file>
      <line>93</line>
    </frame>
    <frame>
      <ip>0x100350DDD</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::vm::VirtualMachine::initialize::{{closure}}</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src</dir>
      <file>vm.rs</file>
      <line>330</line>
    </frame>
    <frame>
      <ip>0x1003507C1</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::vm::VirtualMachine::initialize</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src</dir>
      <file>vm.rs</file>
      <line>378</line>
    </frame>
    <frame>
      <ip>0x10000290B</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython_vm::vm::Interpreter::new_with_init</fn>
      <dir>/Users/sobolev/Desktop/rustpython/vm/src</dir>
      <file>vm.rs</file>
      <line>1972</line>
    </frame>
    <frame>
      <ip>0x10000166E</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython::run</fn>
      <dir>/Users/sobolev/Desktop/rustpython/src</dir>
      <file>lib.rs</file>
      <line>95</line>
    </frame>
    <frame>
      <ip>0x100002368</ip>
      <obj>target/debug/rustpython</obj>
      <fn>rustpython::main</fn>
      <dir>/Users/sobolev/Desktop/rustpython/src</dir>
      <file>main.rs</file>
      <line>2</line>
    </frame>
    <frame>
      <ip>0x100001F5D</ip>
      <obj>target/debug/rustpython</obj>
      <fn>core::ops::function::FnOnce::call_once</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops</dir>
      <file>function.rs</file>
      <line>227</line>
    </frame>
    <frame>
      <ip>0x100003330</ip>
      <obj>target/debug/rustpython</obj>
      <fn>std::sys_common::backtrace::__rust_begin_short_backtrace</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src/sys_common</dir>
      <file>backtrace.rs</file>
      <line>125</line>
    </frame>
    <frame>
      <ip>0x100002773</ip>
      <obj>target/debug/rustpython</obj>
      <fn>std::rt::lang_start::{{closure}}</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src</dir>
      <file>rt.rs</file>
      <line>49</line>
    </frame>
    <frame>
      <ip>0x101269560</ip>
      <obj>target/debug/rustpython</obj>
      <fn>std::rt::lang_start_internal</fn>
      <dir>/rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/ops</dir>
      <file>function.rs</file>
      <line>259</line>
    </frame>
    <frame>
      <ip>0x10000274D</ip>
      <obj>target/debug/rustpython</obj>
      <fn>std::rt::lang_start</fn>
      <dir>/Users/sobolev/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src</dir>
      <file>rt.rs</file>
      <line>48</line>
    </frame>
    <frame>
      <ip>0x1000023A5</ip>
      <obj>target/debug/rustpython</obj>
      <fn>main</fn>
    </frame>
  </stack>
client stack range: [0x1076C8000 0x107EC6FFF] client SP: 0x107EC4298
valgrind stack range: [0x700001BA0000 0x700001C9FFFF] top usage: 9768 of 1048576


Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.

If that doesn't help, please report this bug to: www.valgrind.org

In the bug report, send all the above text, the valgrind
version, and what OS and version you are using.  Thanks.

@sobolevn

Copy link
Copy Markdown
Contributor Author

@sobolevn

Copy link
Copy Markdown
Contributor Author

It should be good now 🎉
Finally!

@sobolevn

sobolevn commented Aug 29, 2021

Copy link
Copy Markdown
Contributor Author

@youknowone I cannot reproduce this failure (https://github.com/RustPython/RustPython/pull/2974/checks?check_run_id=3454164130) locally. Is it related?
But, I don't have the latest macos.

assert BaseException.__new__.__qualname__ == 'BaseException.__new__'
assert BaseException.__init__.__qualname__ == 'BaseException.__init__'
assert BaseException().__dict__ == {}
assert BaseException.__doc__

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these lines deleted?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread vm/src/exceptions.rs
Comment on lines +586 to +583
define_exception! {
PyStopIteration,
PyException,
stop_iteration,
"Signal the end from iterator.__next__()."
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a weak suggestion. If this one would be not a maro_rules but a proc-macro like now, I think we can keep it more structural way for readability and future extension.

/// Signal the end from iterator.__next__().
#[pyexception(name="stop_iteration", base = "PyException")]
struct PyStopIteration {}

This code have a little bit more code but understandable like other common #[pyclass]es

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! 👍

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is problematic to pass __init__ and tp_new this way. Since proc-macro only allows key=string pairs, not key=ident
Writting

#[pyexception(base = "PyException", ctx_name = "stop_iteration", init="import_error_init")]
struct PyImportError {}

fn import_error_init() { ... }

seems a bit off.

Maybe something like:

#[pyexception(stop_iteration, base_exception_new, import_error_init)]
#[pyclass(base = "PyException")]
struct PyImportError {}

?

But, this does not look way better than define_exception! macro right now.

@sobolevn sobolevn Aug 30, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is even more verbose, because pyclass requires name= and module= to be set:

/// Some doc
#[pyexception(stop_iteration, base_exception_new, import_error_init)]
#[pyclass(name = "ImportError", module = false, base = "PyException")]
struct PyImportError {}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • new and init:tp_new and init can be added in pyimpl.
  • #[pyexception] can trigger #[pyclass] inside the macro. So they shouldn't be stacked.
  • I don't think string and ident matter. We already use string literal for pyclass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new and init:tp_new and init can be added in pyimpl.

Do you mean like manually? Or do you mean with some extra macro syntax?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't expect extra macro there. We already have pyimpl, so I thought it could work just like pyclass.

@youknowone youknowone Sep 5, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I expect finally it looks like:

#[pyexception(name = ...)]
struct ImportError {}

#[pyimpl]
impl importError {
    #[pyslot]
    fn new(...) {}

    #[pymethod(magic)]
    fn init(...) {}
}

I don't think declaring a free function import_error_init and use the name in the macro have advantage to the traditional way.

@fanninpm

Copy link
Copy Markdown
Contributor

What about an exception in, e.g., vm/src/stdlib/termios.rs?

@youknowone

Copy link
Copy Markdown
Member

@coolreader18 looks busy.

I left a few suggestions but I think this version is good enough because it is solving problems.
@sobolevn do you have plan for more work? otherwise, could you rebase this PR to merge current version?

@sobolevn

Copy link
Copy Markdown
Contributor Author

@sobolevn do you have plan for more work? otherwise, could you rebase this PR to merge current version?

Yes, I do! But, I don't have the time right now. This requires quite a lot of physical labor 😆
I will just rebase it for now.

@sobolevn sobolevn force-pushed the extends-exception-macro branch from a25f47e to 63db6d7 Compare September 15, 2021 07:35
@sobolevn

Copy link
Copy Markdown
Contributor Author

Rebased! Please, squash it on merge

@youknowone youknowone merged commit b723bbf into RustPython:main Sep 15, 2021
@youknowone

Copy link
Copy Markdown
Member

thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CPython3.8+ does not have TargetScopeError exception

4 participants