Repository navigation
Fix #481 using custom qt4 input hook - #815
Conversation
There was a problem hiding this comment.
Are there cases where neither of these are true? What would happen if so?
Also, a minor nitpick, is it necessary to use os.name for one test and sys.platform for the other? Can we get the information we need from just one?
There was a problem hiding this comment.
(OK, I see now this code is just moved from another file. It's still worth thinking about while we're looking at it, though)
There was a problem hiding this comment.
Well, you have the same if/elif test structure in the stdin_ready() function, so it will just work.
Perhaps it would be cleaner to move those imports in the stdin_ready() function directly?
(if that's not against your coding conventions)
There was a problem hiding this comment.
But if neither of these conditions were true, stdin_ready will always return None, which looks like it would put the code below into an infinite loop (I could be wrong, I don't do much GUI stuff).
There was a problem hiding this comment.
Correct! I don't think there's a supported platform which doesn't fall in either category, but anyway returning True as a fallback seems safer. My next iteration of the function will look like this:
def stdin_ready():
if os.name == 'posix':
import select
infds, outfds, erfds = select.select([sys.stdin],[],[],0)
if infds:
return True
else:
return False
elif sys.platform == 'win32':
import msvcrt
return msvcrt.kbhit()
return True # assume there's something so that we won't wait foreverAlso, I assume that the original combination of os.name/sys.platform was probably done on purpose: match Linux and Mac with the first test, and pick non-cygwin python on Windows with the second test.
There was a problem hiding this comment.
What are os.name and sys.platform inside cygwin? Does the function work properly in this case too?
There was a problem hiding this comment.
For cygwin, os.name is 'posix' and sys.platform is 'cygwin', so Cygwin is handled by the first case, environments which have select.
We could replace the second test with simply os.name == 'nt' if you prefer, the semantic will be the same if the tests are done in that order.
(PS: I haven't tested in cygwin, I don't even know if it's a supported platform!)
There was a problem hiding this comment.
If testing for os.name == 'nt' will behave as expected, I think that's better than using two different variables to test. I've no idea if IPython can run on Cygwin, but if it can, I don't doubt there'll be someone using it.
FWIW, the Python docs describe sys.platform as having a "finer granularity" than os.name.
|
Dealing with CTRL+C proves to be quite uneasy. First on Windows with Python 2.7, I see all the weird effects described in #625. The root cause might well be a Python bug, see e.g. http://bugs.python.org/issue1677 though in my case the python interpreter exits immediately, at the first CTRL+C). So here, at the very least we need to trap KeyboardInterrupt in the hook, like it's done for other gui backends (6d5890c). In 0dd489f, I also played with the idea of breaking the event loop processing with a first press to CTRL+C, which lets a second CTRL+C raise a proper KeyboardInterrupt and makes you return to the prompt. At this point the event processing is restored in the inputhook thanks to a pre_prompt_hook. I find this behavior to be quite OK (kind of fix for #122). But that only works when you start IPython with What I ideally would like to achieve is the 0dd489f behavior but also when starting IPython via |
|
I'm not a Windows user, so I can't help you with the specifics there. I think ipython.exe is something created by Enthought for EPD, so you might want to get in touch with them for how that's put together. |
|
Hey guys, just a quick note: I haven't followed the PR, but right now it doesn't merge cleanly anymore. Sorry about that, there's been enough churn on master lately that some PRs fell behind. So this one needs a rebase before we can proceed further, let us know if you need a hand with that. Unless you guys have decided you want to take a different approach altogether with a new PR, I'll leave that up to you. I just want to give you a heads-up b/c I'm starting to map out the landscape to plan for an 0.12 release, and un-mergeable PRs will be obviously an issue there. |
|
I think a big PR that I merged carelessly may have caused various conflicts, so I'm happy to lend a hand rebasing if you like. |
|
I wanted to rework the changesets anyway to take the feedback into account, so I don't mind rebasing. I'll be able to create a new PR this week-end (or is there a way to associate a second branch or a rebased branch on the same PR?). |
|
It's a bit 'impure', but if you rebase a branch and force-push it (i.e. push the new branch with the old name), it updates the pull request accordingly. We do that quite a bit. |
|
Yup, don't worry about rebasing and doing |
Move it from lib.inputhookwx to lib.inputhook, as the former module can only be loaded if wx itself is available.
- move select and msvcrt imports in the function itself as they're only needed there - switch on os.name for platform dependent code - for unknow platforms, assume there's something to read
PyQt4's own input hook is not really appropriate when PyReadline is active (and possibly other readline packages running the PyOS_InputHook repeatedly). Arguably this should be fixed in PyQt4 itself, but in the meantime we can install a local hook which does the Qt event loop processing in a less aggressive way.
Allow a first CTRL+C to interrupt the qt4 event loop and a second to trigger a KeyboardInterrupt.
…dline'. While the previous code works fine on Windows, on Linux with 'readline' the situation is different: once we disable the Qt4 event loop, we go back waiting in the select() inside readline.c's readline_until_enter_or_signal() function. When that select() is interrupted, the PyOS_InputHook is executed if present and the pending KeyboardInterrupt is delivered as soon as bytecode execution starts, even before entering the `try:` clause! As at that point we're in a ctypes callback, this will lead to an internal error. The only solution is therefore to disable the PyOS_InputHook temporarily.
Second iteration of the patchset.The first 3 changesets deal with #481:
The last 2 changesets deal with the Qt part of #122:
The last two are not needed to fix #481, but without them the PyQt4 support seems buggy, as explained in #122. As explained in the changesets themselves, I think there are a few technical limitations in ctypes that prevent a single CTRL+C to work as expected, hence the "two CTRL+C" approach which I believe is intuitive enough, even more so now that I print after the first CTRL+C: Open points:
I think only 1. prevents merging, the others could correspond to later improvements. |
There was a problem hiding this comment.
All functions should have a docstring. The full docstring guidelines follow the numpy standard, but for such a small function probably a single one-liner suffices.
Though in this case, I'd prefer it if the function was split in three: one for posix, one for nt, and an 'others'. The if statement that's currently inside would then instead assign the public name to the right implementation. This raises slightly the import cost, but means we're not doing platform checks on every single call.
|
Thanks a lot for the updates. I had some inline comments, once those are handled we'll play further with this guy, it looks like we're getting close to being able to merge it, and it should be in time for 0.12. |
- split stdin_ready() in 3 platform specific functions and choose the appropriate one at import time - add docstrings for each of these
|
I agree that enable_qt4() was a bit too long and needed clean-up. However, I think it makes sense to keep inputhook_qt4() and preprompthook_qt4() as subfunctions, so that they can share a variable from a parent function |
- make sure we don't install more than one 'pre_prompt_hook' for a given ipapi instance - clarify the internal documentation and move it from comments into docstrings
|
I was a bit worried about the use of a 'pre_prompt_hook', as I was not completely sure about the lifetime and scope of the objects involved (is Btw, it seems that there's no way to get |
|
On Mon, Oct 10, 2011 at 5:43 AM, Christian Boos
Yes, it's a singleton, though now it's spelled get_ipython(); we've
That sounds like a good idea, yes. If we're making %gui a user-facing |
|
While working on the open point 2. auto-repeat of CTRL+C leads to an internal error, I found a solution which implies some cooperation between set_inputhook, clear_inputhook and the actual callback (inputhook_qt4). It works fine, I no longer have the internal error, but the problem is that the other GUIs will not see a KeyboardInterrupt anymore. It's not that critical in the sense they were just ignoring it anyway or not handling it at all (cf. #122, #825), but to me this means we can probably do better and extend my two step approach (first CTRL+C to disable the event loop, second to trigger a normal KeyboardInterrupt) to all GUIs, not just Qt... I think this will lead to a cleaner approach overall. If in the meantime someone could test the current state on Mac, this would be interesting as well ;-) |
On systems with 'readline', it's very likely to intercept a signal during a select() call. The default SIGINT handler will schedule a KeyboardInterrupt exception to be raised as soon as possible. If ctypes is used to install a Python callback for PyOS_InputHook, this will happen as soon as the bytecode execution starts, so even if the first instruction of the callback is a `try: ... except KeyboardInterrupt` clause, it's actually too late. As ctypes doesn't allow a Python callback to raise an exception, this ends up with IPython detecting an internal error... not pretty. We must therefore ignore the SIGINT signals until we are sure the exception handler is active, in the Python callback.
|
Okay... so here it goes ;-) The last two commits lead to a result I'm much more satisfied with. It also feels more reusable for other GUIs. They are normally not affected by the change, as the set_inputhook() keeps the current semantic and I've introduced a set_safe_inputhook for the new behavior, so far only used for qt4. If you look at this PR now, you may want to skip most of the intermediate changes and look at the final inputhook.py and inputhookqt.py files, I think I've documented everything quite clearly ;-) If you like the approach, I could try to rebase once more and go to this approach directly. |
|
Thanks for testing on Mac! Both the original error and the one in my hook have a common cause: assuming the qApp is still alive, which for some reason is not the case here. I'll try to reproduce that issue. I can check for this situation in the qt4 hook and as you suggested make sure that the hook will unregister itself in case of an error. Also in parallel to this set of changes, I continued the work started in 1d6e9a0 for using the two step CTRL+C approach for all GUIs, and some initial testing has shown that this approach can also work well for gtk and Tk. When this is ready I'll create a second PR in order to keep this one focused on the immediate problem with Qt4 (i.e. fixing #481). |
|
That makes sense. Can you see why the atexit behavior is not the same with the new code? |
|
I installed PySide on Windows (the latest 1.0.7 installer) and I could reproduce the problem... only when I mistakenly mixed PyQt4 and PySide code! They seem to both share the same instances of Qt libraries and to interfere badly at exit. Are you sure you don't have a If I only use PySide and don't mix it with PyQt4, then everything works fine (#481 also fixed, no crash at exit), at least on Windows. Next step is to test this in EPD on Windows with matplotlib, and then on Linux... |
- return early if there's no longer a qApp - trap any exception that could be raised from within the hook, report it and unregister the hook
|
Good find, but I am sure that I am not importing both PyQt and PySide, because I don't have both on this machine. Base on the MPL discussion, it seems like this might be a PySide bug, but there's clearly a way to avoid it, because it doesn't affect IPython master, it only comes up using this PR. |
|
Ok, which version of matplotlib were you using? |
|
git master |
|
Ok, I can now also reproduce the above crash on Windows, with matplotlib 1.2.x (2bb91ce). But even with ipython master (cb6da34) I get a similar crash at exit: Besides, with ipython master there's no event loop integration with PySide. You can however do I think that having the crash or not depends on the order in which the atexit handlers are run. I use a VisualStudio 2008 build, and perhaps you have a mingw one? In any case, relying on the order of atexit handlers is fragile at best. I see that matplotlib in backend_qt4.py tries to detect a Bottom line: it's a matplotlib issue due to PySide behaving differently than PyQt4, nothing really caused by the changes here. Quite the opposite, PySide is better supported as you get event loop integration and with my last commit da13a62, I handle this crash in a more robust way by unregistering the hook. I get a full crash report however, instead of only the backtrace as above, for some reason. Suggested fix for matplotlib: diff --git a/lib/matplotlib/backends/backend_qt4.py b/lib/matplotlib/backends/backend_qt4.py
index c85b4b3..64a737e 100644
--- a/lib/matplotlib/backends/backend_qt4.py
+++ b/lib/matplotlib/backends/backend_qt4.py
@@ -382,6 +382,7 @@ class FigureManagerQT( FigureManagerBase ):
self.window.show()
def destroy( self, *args ):
+ if QtGui.QApplication.instance() is None: return
if self.window._destroying: return
self.window._destroying = True
QtCore.QObject.disconnect( self.window, QtCore.SIGNAL( 'destroyed()' ),
|
|
Wow, I don't know how I managed that bad report. You are right, master doesn't work at all in PySide. And I do know the reason that it's a crash in your branch, and a traceback in master. I fixed that bug last week: d5548fa If you rebase on master (I just did, it went cleanly), you will just get a traceback. I'm fine with this, then, as it is indeed a matplotlib and/or PySide bug, and doesn't crash IPython. I recomend sending your fix upstream to matplotlib, as it seems clean. |
|
I've been able to test this branch with PyQt, PySide, Tk, Wx, Gtk on Windows and Linux, and also with pylab on both platforms. The test suite passes on both platforms as well. mrink tested pylab with pyside for the Mac. Now all looks good i.e. #481 is fixed for PyQt4 and PySide, no ill side-effects... Is there anything else I should do? I wonder if I should rework the patchset once more in order to have fewer and only "clean" changesets inside. And one thing I'm unsure about is the impact on py3k support... I don't know exactly what the current status is supposed to be, but I couldn't get master to work with Python 3.2.2. Of course there are still a lot of improvements that could be made in the IPython/lib/inputhook* area, especially now that I've tested more GUI backends and I see that they could also benefit from this new approach. But I'll save this for another PR (continuing on http://github.com/cboos/ipython/commits/inputhook-refactoring). |
|
Python 3 support should be working in master. What problems are you seeing? I'll grab a fresh checkout and have a look. |
|
Great, let's wait for @takluyver's report on py3 before we make any final decisions on this one. But it's looking very close to ready for merge, thanks @cboos for the work and patience! |
|
Master seems to be working fine in Python 3.2.2 (give or take #939, which I've just opened, but that's a minor problem). I don't think I've experienced the problem this PR aims to solve, so I'm not sure if I can usefully test it. |
|
@takluyver, testing this amounts to going to Since I don't know whether we've actually tested this or not with Py3 before, I suggest running those tests first in master and then with @cboos PR merged in. We should only bother this PR if there's a difference after merging it; any problems that may exist in master we'll ignore for the purposes of this PR an deal with separately. |
|
Qt and Tk appear to be working fine, both in master and this branch. Gtk and Wx aren't on Python 3 - in fact, Gtk is, but I think our integration is for PyGTK, not the new gobject bindings. |
|
OK, I've tested this as much as I can interactively, and given that the rest of us here have also run it through its paces, I'll merge it now. There may be lurking issues (the GUI integration stuff is devilishly hard) but we won't find those until we put it to broader, real-world usage. @cboos, many thanks for this! I know it was hard work with a lot of back and forth, so I'm glad you stuck with it. It's a necessary hurdle to keep the code quality as high as we can given the small size of our team. |
Fix #481 using custom qt4 input hook. Various other cleanups to inputhook libraries.
|
@ALL: thanks for testing and merging! @takluyver: regarding py3k, |
|
@cboos: You have to do |
|
@takluyver: Ok, got it! Thanks for pointing me in the right direction. It's not yet fully working (compat issues with pyreadline 2.0), but at least now it make sense! |
Fix ipython#481 using custom qt4 input hook. Various other cleanups to inputhook libraries.
Don't ignore ctrl-C during `%gui qt` PR ipython#815 ignores ctrl-C during normal code execution, this restores the sigint handler outside the inputhook. Closes ipython#1157.
This is my tentative fix for issue #481, as discussed in #481 (comment).
This is not meant to be merged as such, as I only could provide limited testing:
Also, I've currently inlined the hook, as it was small enough it didn't feel appropriate to create a IPython/lib/inputhookqt.py file.
Finally, the behavior with Ctrl-C is currently quite catastrophic, though this is not related to my changes, as it's the same behavior with the original 0.11 install, even without Qt involved. But this probably another topic.
See second iteration for the merge candidate.