Thanks for reviving that.

1. I believe in 3.7 await at top level is valid; should we support that only for 3.7+, and get rid of all the AST munging and complexity of making the parsing work in 3.6 and below ?

2. I'm baffled by the test faillures iptest IPython.core.tests.test_application IPython.core.tests.test_async_helpers fails, but each independently succeed.

I'm baffled by the test faillures iptest IPython.core.tests.test_application IPython.core.tests.test_async_helpers fails, but each independently succeed.

No, my bad, just seem to be random. test_async_helpers fails on its own.

I believe in 3.7 await at top level is valid

I don't think this is true, unfortunately. Maybe 3.8 if someone steps up to do it; I suspect the main asyncio maintainers will be too busy to get to it.

@minrk obviously tornado is the natural place to start given how ipykernel works, but have you thought at all about how to support alternative event loops?

I don't think this is true, unfortunately. Maybe 3.8 if someone steps up to do it; I suspect the main asyncio maintainers will be too busy to get to it.

Well, ok, maybe 3.8, but in nightly at least ast.parse now allows async top level.
compile does fail though.

@minrk obviously tornado is the natural place to start given how ipykernel works, but have you thought at all about how to support alternative event loops?

That is also my concern, but then I guess we have 2 questions:
Do we allow to run trio/curio code by locally blocking for the current cell ?
Because when running in ipykernel, you obviously can't run using the trio or curio runner to run async_cells. Or can we ?

I've pushed a tiny fix to the loop_runner to ensure it is a callable, and that if curio,trio or asyncio string were passed they were mapped to the correct runner, and not to the module themselves after imports.

With this running IPython under trio does seem to work, with the asyncio loop not running. Not sure what that does in a notebook though.

Ok, one of the error left is in nesting IPython, as the event loop is already running:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-0c5ef5f92d7e> in <module>
----> 1 IPython.embed()

~/dev/ipython/IPython/terminal/embed.py in embed(**kwargs)
    384         frame.f_code.co_filename, frame.f_lineno), **kwargs)
    385     shell(header=header, stack_depth=2, compile_flags=compile_flags,
--> 386           _call_location_id='%s:%s' % (frame.f_code.co_filename, frame.f_lineno))
    387     InteractiveShellEmbed.clear_instance()
    388     #restore previous instance

~/dev/ipython/IPython/terminal/embed.py in __call__(self, header, local_ns, module, dummy, stack_depth, global_ns, compile_flags, **kw)
    227         # our call and get the original caller's namespaces.
    228         self.mainloop(local_ns, module, stack_depth=stack_depth,
--> 229                       global_ns=global_ns, compile_flags=compile_flags)
    230
    231         self.banner2 = self.old_banner2

~/dev/ipython/IPython/terminal/embed.py in mainloop(self, local_ns, module, stack_depth, display_banner, global_ns, compile_,flags)
    322
    323         with self.builtin_trap, self.display_trap:
--> 324             self.interact()
    325
    326         # now, purge out the local namespace of IPython's hidden variables.

~/dev/ipython/IPython/terminal/interactiveshell.py in interact(self, display_banner)
    474             else:
    475                 if code:
--> 476                     self.run_cell(code, store_history=True)
    477
    478     def mainloop(self, display_banner=DISPLAY_BANNER_DEPRECATED):

~/dev/ipython/IPython/core/interactiveshell.py in run_cell(self, raw_cell, store_history, silent, shell_futures)
   2794                 store_history=store_history,
   2795                 silent=silent,
-> 2796                 shell_futures=shell_futures,
   2797             )
   2798         )

~/dev/ipython/IPython/core/async_helpers.py in _asyncio_runner(coro)
     23     """
     24     import asyncio
---> 25     return asyncio.get_event_loop().run_until_complete(coro)
     26
     27

~/anaconda/lib/python3.6/asyncio/base_events.py in run_until_complete(self, future)
    452         future.add_done_callback(_run_until_complete_cb)
    453         try:
--> 454             self.run_forever()
    455         except:
    456             if new_task and future.done() and not future.cancelled():

~/anaconda/lib/python3.6/asyncio/base_events.py in run_forever(self)
    406         self._check_closed()
    407         if self.is_running():
--> 408             raise RuntimeError('This event loop is already running')
    409         if events._get_running_loop() is not None:
    410             raise RuntimeError(

RuntimeError: This event loop is already running

In [4]: ---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-4-428746cf6514> in <module>
----> 1 print('true' if embed1 is not ip0 else 'false')

NameError: name 'embed1' is not defined

That, I'm also not sure how to fix.

So one possibility is to run the nested instances using different event loop; or to drop the support for nested embed (for now?) until all is async and we can run an async embed.

If I understood correctly, from the work @Carreau did with the loop_runners, this should work with any runner as long as that runner can run async def coroutines, which is a reasonable requirement, right?

Now, what happens in a case like ipykernel where tornado (and therefore also asyncio) is already running, I'm not sure how to deal with that. Right now, ipython/ipykernel#323 unconditionally uses run_cell_async and the existing asyncio runner. We could try to handle the cases where the requested runner is not the already-running one, and do a synchronous run with the chosen eventloop.

i.e. run_cell_async could do something like:

if loop_running:
    return coro() # will be awaited outside
else:
    return loop_runner(coro())

@Carreau re: nested IPythons, I'm really not sure how to solve that one. I made this a blocking wrapper around a fundamentally async core because it was the only way to get it to work without duplicating all of the methods as async and non-async variants all the way down. asyncio explicitly doesn't do nested eventloops, so too allow nested IPythons with blocking calls, we would have to make sure that we never instantiate an eventloop. Similarly, to allow nested IPythons with async, we would need to allow embedding IPython in a running eventloop, which should be quite doable (ipykernel is already doing this by simply calling run_cell_async instead of run_cell).

Or we can try to violate the "no nested eventloops" assumption of asyncio by handling our own stack of clearing/replacing the current loop.

I'm happy to drop the necessity to have nested IPython's but we should likely ask on the mailing list first for how many users of this feature we have.

I think the need for embed() have two sides:

1. Running in an existing eventloop
2. Blocking an event loop and inspecting current state.

I don't believe in (2) we need a lot of async integrations so I'm happy to clear/replace eventloop.

@njsmith how hard would it be to have a version of trio.run that can be reentered ? How negatively would you react to us wrapping trio.run in the following context manager ?

@contextmanager
def new_context():
    import trio._core._run as tcr
    old_runner = getattr(tcr.GLOBAL_RUN_CONTEXT, 'runner', None)
    old_task = getattr(tcr.GLOBAL_RUN_CONTEXT, 'task', None)
    if old_runner:
        del tcr.GLOBAL_RUN_CONTEXT.runner
    if old_task:
        del tcr.GLOBAL_RUN_CONTEXT.task
    yield
    if old_runner:
        tcr.GLOBAL_RUN_CONTEXT.runner = old_runner
    if old_task:
        tcr.GLOBAL_RUN_CONTEXT.task = old_task

AFAICT this commit make multiple nested embedded IPython's works (using trio, and above context manager, and relying on trio's internal). You can (of course) make IPython crash if you try to switch to asyncio loop in one of the nested embedded ones.

cc @mrocklin, would that be of any use when interacting with dask interactively ? See ipython/ipykernel#323 as well when running that inside a notebook.

There is a significant amount of difference with master now and rebase is tough. In order to keep some history I'm going to close and open a PR where all the work as been merged into a single commit and the merge commits from this removed.

see #11265