Skip to content

Refactor kernel channel classes - #7108

Merged
minrk merged 31 commits into
ipython:masterfrom
takluyver:digging-channels
Dec 11, 2014
Merged

minrk merged 31 commits into
ipython:masterfrom
takluyver:digging-channels

Conversation

@takluyver

Copy link
Copy Markdown
Member

The kernel 'channels' system had grown into a horribly tangled thicket of multiple inheritance code, using inheritance for three orthogonal differences: channel kind, blocking vs Qt, and ZMQ vs in-process. This is my attempt to trim it into some kind of clarity before we split the repos up (which will make doing this much harder).

  • The shell, iopub and stdin channels are no longer distinguished by different classes; they are all instances of the same class. When used with ZMQ, their socket is created and connected for them, and passed in. The exception is the BlockingInProcessStdInChannel, which remains a separate class for now.
  • Heartbeat channels no longer share inheritance with the other kinds of channels, as they're performing a rather different function.
  • Channels (from here meaning shell/iopub/stdin) are not threads in any case. For the Qt+ZMQ case for which channels were made threads, there is a new IOLoopThread class, which runs a single ioloop in a thread and handles sending and receiving messages on all three channels.
  • The methods to send specific message types (e.g. execute() and inspect()) have all been moved from the channels to the KernelClient. They were already available on the kernel client by some proxying magic, but now they are actually define on the client class.
  • The Qt channels had a bunch of signals emitted when specific message types were received, but the dispatch logic used the message_received event, and then looked at the message type again, so all the specific signals were unused. I have therefore removed them to make it more obvious what's going on.

Doing this has allowed me to cut out lots of complex class inheritance and mixins. I briefly thought about trying to leave things in place for backwards compatibility, but I don't think there is any sane way to do that in this case. Hopefully, most consumers deal with the KernelClient class, which is largely unchanged.

@dsblank

dsblank commented Dec 5, 2014

Copy link
Copy Markdown
Contributor

As an outsider attempting to muck with IPython occasionally, I have found myself attempting to follow that rats nest a few times. It would be great to see that refined sooner than later. Thanks!

@Carreau

Carreau commented Dec 5, 2014

Copy link
Copy Markdown
Member

We should ping someone at Enthought too - who do we usually contact there?

I would say we have the choice between :

  • epatters
  • rkern
  • jonathanrocher
  • jdmarch

I'll let you pick.

@takluyver

Copy link
Copy Markdown
Member Author

I'll just ping all four, and let them pick ;-)

@epatters @rkern @jonathanrocher @jdmarch - this PR involves a substantial refactoring of the 'channel' classes used internally in the Qt console. I hope no code outside IPython is using these classes directly, but if it is, please let us know. Other code working with the KernelClient API may be affected, but it should be fairly straightforward to fix that.

@jdmarch

jdmarch commented Dec 5, 2014

Copy link
Copy Markdown

@takluyver thanks for the heads up!

@ccordoba12

Copy link
Copy Markdown
Member

I also agree with @dsblank: this should go in 3.0 rather than later because these are important API changes. I'll give some look to the Qt part next week.

Comment thread IPython/kernel/blocking/channels.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is all a channel really does now, it probably makes sense to replace ._queue_send with .send.

@minrk

minrk commented Dec 8, 2014

Copy link
Copy Markdown
Member

I've had a read through, and this seems like a solid improvement over the current situation. A refactor like this close to release does make me a bit uneasy, but as has been discussed above, doing it now will be a lot less painful than doing it after the repos separate. Other than a minor pyzmq style point above, I'm 👍 to all this.

@takluyver

Copy link
Copy Markdown
Member Author

Addressed Min's comments.

@Carreau

Carreau commented Dec 9, 2014

Copy link
Copy Markdown
Member

as has been discussed above, doing it now will be a lot less painful than doing it after the repos separate.

+1 then.

@minrk

minrk commented Dec 9, 2014

Copy link
Copy Markdown
Member

I've looked it over, and it all seems to make sense to me. I'll wait for @ccordoba12 to do a sanity check on the Qt parts, then I'd say it's okay to merge.

@bfredl

bfredl commented Dec 9, 2014

Copy link
Copy Markdown
Contributor

These are nice clean ups, but I wonder this will affect third-party console clients that want to communniucate with a ZMQ kernel asyncronously? The following is probably not The Right Way To Do It, but in a client for IPython 2.x (an experimental plugin for Neovim) I do the following to recieve async notifications for messages:

class IPythonVimApp(BaseIPythonApplication, IPythonConsoleApp):
    kernel_client_class = KernelClient
    def init_kernel_client(self):
        self.kernel_client = self.kernel_manager.client()
        self.kernel_client.shell_channel.call_handlers = self.target.on_shell_msg
        self.kernel_client.iopub_channel.call_handlers = self.target.on_iopub_msg
        self.kernel_client.stdin_channel.call_handlers = self.target.on_stdin_msg
        self.kernel_client.hb_channel.call_handlers = self.target.on_hb_msg
        self.kernel_client.start_channels()

    def initialize(self, target, argv):
        self.target = target
        super(IPythonVimApp, self).initialize(argv)
        IPythonConsoleApp.initialize(self, argv)

(where the target, running inside a different event loop, is aware notifications come from a different thread).

Now how will this the simplest be done on the new infrastructure? To me it seems the new QtKernelClient and its helper classes does almost what another async console client wants (or at least mine); I mean very few parts of it are Qt specific. Do you think it would make sense to later refactor this into a generic async/thread-safe zmqsocketchannel/kernelclient (and perhaps implement the Qt kernel client on top of it)?
If it would be interesting, I could try implement this once this PR gets merged.

@takluyver

Copy link
Copy Markdown
Member Author

Thanks @bfredl for that use case.

Yes, it may well at some point make sense to split out a base class for a generic async client. I'd rather not do that here, because it's a big enough PR already, and since we only have one asynchronous client (the Qt console), I probably wouldn't get the abstractions right.

There are really two possibilities for async clients:

  • Run the pyzmq event loop in a thread and use some mechanism to go between that and your main event loop. This is what I've made the Qt console do for now, but it's the quick and dirty option.
  • integrate listening to the ZMQ sockets into your main event loop. This is a bit more work, but @minrk assures me that it is practical.

@bfredl

bfredl commented Dec 9, 2014

Copy link
Copy Markdown
Contributor

@takluyver Yes, ideally, a base client class could be able to either provide a list of ZMQ fds (and callbacks for each) to the client developer to plug in to a event loop framwork of their choosing, or spawn a thread if preferred, but it probably requires some work to be thread-safe in all cases and have suitable abstractions.

@ccordoba12

Copy link
Copy Markdown
Member

Thanks for waiting for me @minrk and @takluyver. I'll do my review tomorrow, I was just fixing some breakage I introduced in Spyder yesterday :-)

@minrk

minrk commented Dec 10, 2014

Copy link
Copy Markdown
Member

@ccordoba12 No worries, thanks for reviewing!

@takluyver

Copy link
Copy Markdown
Member Author

I ran our test suite against this on Windows, and it passed on Python 3.4 and Python 2.7.

Comment thread IPython/qt/client.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These make_*_socket functions doesn't seem to be used in this file. Are they really necessary?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spoke to Min, and they are redundant with some methods on ConnectionFileMixin. Removing, thanks.

@ccordoba12

Copy link
Copy Markdown
Member

Other than the little correction I suggested, I think the rest is fine by me. I can't test it with Spyder because of the changes to QtKernelClient but the design is a lot cleaner and easier to follow than before. Thanks @takluyver for the effort! :-)

@minrk

minrk commented Dec 11, 2014

Copy link
Copy Markdown
Member

Great, thanks @ccordoba12.

We can check in at the meeting this morning, and merge today.

minrk added a commit that referenced this pull request Dec 11, 2014
@minrk
minrk merged commit fe5dd27 into ipython:master Dec 11, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants