Whoooo ! Will review !

So before reviewing I'll still ask: It seem like PTShell make it more difficult (impossible) to us this in things like emacs inferior shell. It will likely not be the only one, so should we really get rid of the readline one ?Or leave it as an option ?

So before reviewing I'll still ask: It seem like PTShell make it more difficult (impossible) to us this in things like emacs inferior shell. It will likely not be the only one, so should we really get rid of the readline one ?Or leave it as an option ?

+1 otherwise.

• Do we have an issue or something about use in emacs? How bad is it, and is there any chance things could get fixed?
• What is emacs inferior shell doing/not doing that affects this? Do we know of any other examples that will be affected? Early on we thought that we might need the readline one for use over SSH, but prompt_toolkit turned out to work fine.

I'm concerned that if we have two sets of terminal interaction code and are mainly using one, it will bit rot. Maybe we could split it out and have it as a separate package for emacs users to maintain? But the APIs the InteractiveShell subclasses use are not very well defined, so I'm not really keen on that either.

Do we have an issue or something about use in emacs? How bad is it, and is there any chance things could get fixed?

ping @tacaswell

I'm concerned that if we have two sets of terminal interaction code and are mainly using one, it will bit rot. Maybe we could split it out and have it as a separate package for emacs users to maintain? But the APIs the InteractiveShell subclasses use are not very well defined, so I'm not really keen on that either.

Fair enough. We can always resurrect the code from git right ? That's what the git dig command from the git-archaeologist plugin is for ?

I am motivated to look in to this (and suspect that it can be fixed on the emacs side) and if not then on the pt side to not emit some escapes.

Emacs chokes in a function that converts ansi escapes into emacs font faces.

This is 90% conjecture.

Great, I'm glad someone has the motivation to investigate. We'll put this on hold for the moment then to see if you can work the issue out. If it's not possible, we'll work out what we're going to do.

@tacaswell have you had a chance to look at this?

A bit but not enough to resolve it.

On Tue, May 10, 2016, 06:18 Thomas Kluyver notifications@github.com wrote:

@tacaswell https://github.com/tacaswell have you had a chance to look
at this?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#9399 (comment)

@tacaswell any more updates? I'd like to get this code out before we release 5.0.

I'd like to get this code out before we release 5.0.

I would be +1 for merging, we can revert if really necessary, but I don't see us reverting PTK for now.

between day-job and mpl 1.5.2/2.0 I have had no time 😞

When is 5.0 slated to be out?

attn @jorgenschaefer who might be able to tell me that is is just a minor configuration thing.

I do not use ipython, so no idea about any of this, sorry. Emacs' python.el has some hardcoded stuff in python-shell-completion-native-setup that tries to use readline or rlcompleter in turn. I don't know at what point it fails, but the correct fix here would be (IMHO) to fix Emacs.

Though, considering the release cycle, that can take a while.

For the curious, the Python code Emacs uses: https://github.com/emacs-mirror/emacs/blob/master/lisp/progmodes/python.el#L3206-L3254

between day-job and mpl 1.5.2/2.0 I have had no time

I hear you.

When is 5.0 slated to be out?

I'd like to have it out for end of June, to have pushed in conad for SciPy.
As it will be LTS, I'd like to release a beta relatively soon to get things battle tested.

I do not use ipython, so no idea about any of this, sorry. Emacs' python.el has some hardcoded stuff in python-shell-completion-native-setup that tries to use readline or rlcompleter in turn. I don't know at what point it fails, but the correct fix here would be (IMHO) to fix Emacs.

Thanks for pitching in. Will try to get a few emacs-users to pitch in, will ping people on ML, and twitter.

Does this remove the dependency on gnureadline on Macs? That would be nice :-)

Yep, and the dependency on pyreadline on Windows :-)

@jorgenschaefer The issue is not with calls into ipython, it is with colorizing the string that comes back from ptprompt

Debugger entered--Lisp error: (args-out-of-range "�[8D�[0m�[J�[0;38;5;28mIn [�[0;38;5;10;1m1�[0;38;5;28m]: �[8D�[0m
�[J�[0m" 80)
  string-match("�" "�[8D�[0m�[J�[0;38;5;28mIn [�[0;38;5;10;1m1�[0;38;5;28m]: �[8D�[0m
\n�[J�[0m" 80)
  ansi-color-filter-apply("�[?25l�[8D�[0m�[J�[0;38;5;28mIn [�[0;38;5;10;1m1�[0;38;5;28m]: �[8D�[0m
\n�[J�[0m�[?7h�[?12l�[?25h")
  python-shell-comint-watch-for-first-prompt-output-filter("�[?25l�[8D�[0m�[J�[0;38;5;28mIn [�[0;38;5;10;1m1�[0;38;5;28m]: �[8D�[0m
\n�[J�[0m�[?7h�[?12l�[?25h")
  run-hook-with-args(python-shell-comint-watch-for-first-prompt-output-filter "�[?25l�[8D�[0m�[J�[0;38;5;28mIn [�[0;38;5;10;1m1�[0;38;5;28m]: �[8D�[0m
\n�[J�[0m�[?7h�[?12l�[?25h")
  comint-output-filter(#<process Python> "�[?25l�[8D�[0m�[J�[0;38;5;28mIn [�[0;38;5;10;1m1�[0;38;5;28m]: �[8D�[0m
\n�[J�[0m�[?7h�[?12l�[?25h")

Which given how badly gh is mangling this copy/paste makes me assume it is an encoding issue.

Looks like a bug in ansy-color-filter-apply, again something to report to the Emacs maintainers :-)

@tacaswell As a workaround, you can probably use %colors NoColor before execution the code which has the problem... Or is that problem happinging in lisp code? Can't actually read/parse it :-)

I think we add the discussion on twitter, and as far as I can remember it's a question of inferior-shell, vs ansi-terminal:

M-X ansi-term

M-x shell

I know there is a reason why you can't use ansi-term, but I'm not sure I remember why.

Also as we see on the screenshots, the color does works, it's so extra escape sequences that are broken, so I'm unsure NoColor is a solution.

[edit] I have no clue what I am doing above.

The reason I use shell instead of ansi-term is that in what elpy-shell-switch-to-shell (eventually) delegates to (hence why I tried to get @jorgenschaefer interested as I suspect that elpy will be getting this bug report in the near future).

The reason I use shell instead of ansi-term is that in what elpy-shell-switch-to-shell (eventually) delegates to (hence why I tried to get @jorgenschaefer interested as I suspect that elpy will be getting this bug report in the near future).

Ah, yes I'm starting to remember.

Then could elpy be made configurable to use ansi-term ? Or is there a reason not to ?

(https://www.masteringemacs.org/article/running-shells-in-emacs-overview)

@takluyver would one option be to "extract" the readline TIS thing into a separate package and "offer it to the community" [Yes I know this solution sucks is not optimal].

It is difficult to interact with ansi-term from Emacs proper. All the REPL modes use comint.

(In theory, ipython should check termcap/terminfo capabilities and not send colors or any terminal commands at all – os.environ["TERM"] == "dumb".)

So technically it's not color, it's the cursor movement sequences and blinking (plus a few other I haven't tracked down yet) , and as this is mostly how prompt_toolkit does it thing, it will likely be super hard to avoid.

Looking a bit at it, it should be (relatively) straightforward to export the "old" terminal interactive shell out of there into a separate package.

Tell me if I'm wrong but M-x shell seem to not handle forward some things like tab completion (do elpy does it through readline ? ) in which case we should be able to reduce that even further.

That would (at least) give some fallback for the time being until we figure out a better plan.

As I'm unsure about how elpy-shell-switch-to-shell is used, I have difficulties understanding what feature is needed, what behavior is expected.

It's certainly possible to split out the old TIS as a separate package, but I'm wary of doing that, because:

1. It creates some kind of expectation that it's supported and maintained. We don't want to do that, and it's not clear that anyone else is willing to put the time in for it.
2. Making it a separate package, especially if other people are maintaining it, relies on the inheritance API between InteractiveShell and TerminalInteractiveShell being stable and well defined, and it isn't. Especially at the moment, we're changing that API quite a bit to improve things with prompt_toolkit, and quite probably breaking/degrading things for the old TIS.

I don't know what the answer is here, but I'm really not keen on the idea of a separate readline TIS package.

There is a very dumb fallback mode used for tests, which just uses Python's raw_input. But you're no better off with that than with the vanilla Python shell.

here is a very dumb fallback mode used for tests, which just uses Python's raw_input. But you're no better off with that than with the vanilla Python shell.

That untrue for magics, ls, and all the syntax on top of Python.
As far as I understand the elpy-shell-switch-to-shell is mostly used to get code executed and get the result back (I might be wrong), so no need for completion, history, and all this fancy stuff. But if that's the case, we can likely rely on a much, much simpler interface than PTK.

And most importantly (from my point of view) the gui event-loop integration management.

Have a go with the environment variable IPY_TEST_SIMPLE_PROMPT=1, and see how that goes. The event-loop integration has all been rewritten (and greatly simplified) for prompt_toolkit, though, so the old inputhook machinery is part of what we'd really like to drop.

One option that might work for a community maintained package is to clean up jupyter_console 4.x, rename that and push it out for whoever's interested in maintaining it - that way it's built on a clearly defined interface (the messaging protocol), and it can take advantage of the event-loop integration in the kernel. jupyter_console 5.x is also based on prompt_toolkit, but 4.x uses readline.

(we could call it 'jupyter inferior shell', which would be accurate in more ways than one ;-)

Using IPY_TEST_SIMPLE_PROMPT=1 appears to fix things for me.

Colors are off from what they were, but I suspect that I can find the right
knobs in emacs to fix that up.

On Thu, May 26, 2016 at 5:37 PM Thomas Kluyver notifications@github.com
wrote:

One option that might work for a community maintained package is to clean
up jupyter_console 4.x, rename that and push it out for whoever's
interested in maintaining it - that way it's built on a clearly defined
interface (the messaging protocol), and it can take advantage of the
event-loop integration in the kernel. jupyter_console 5.x is also based on
prompt_toolkit, but 4.x uses readline.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#9399 (comment)

Have you tested with event loop integration? I'd be surprised if that works with IPY_TEST_SIMPLE_PROMPT. Prompts won't have colour at all, and there's probably no tab completion, unless emacs is doing some magic to provide completions separately.

It looked like the event loop integration worked (had prompt back, could pan/zoom).

emacs does it's own colorization, history, and if set up correctly, completion (via jedi).

It looked like the event loop integration worked (had prompt back, could pan/zoom).

emacs does it's own colorization, history, and if set up correctly, completion (via jedi).

Does multiline input works ?
If so, should we expose that through a --minimal flag or similar ? And then move forward ?

(Sidenote, thanks all for pitching in and helping with this)

multiline does not work, but it never worked very well before (did not cope with the indents right and multi-line scroll back did not work). For multi-line stuff it is better to write it in another buffer and ship it over to the interpreter anyway.

Completion seems to work better now.

Tell me if I'm wrong but M-x shell seem to not handle forward some things like tab completion

It is not a terminal emulator. It treats the subprocess as something that simply emits line-based text to stdout and reads line-based text back. The subprocess should not treat it as a terminal emulator, and should not send terminal commands. This is why you check $TERM before doing these sorts of things.

As I'm unsure about how elpy-shell-switch-to-shell is used, I have difficulties understanding what feature is needed, what behavior is expected.

It's just a wrapper around standard Emacs' run-python, which is the default way of handling interactive programs from within Emacs (same thing for Scheme, Ruby, JavaScript, etc. REPLs). For many of those, Emacs adds features ontop of the underlying REPL. IPython is very featureful by itself, and I don't think it really matches the use case. You are right that using IPython within this environment does not add a lot ontop of standard CPython, but it seems quite a few users like it.

There's an IPython/Jupyter notebook implementation for Emacs which might be more useful for some use cases.

(Again, I do not use IPython at all, so I am just trying to answer questions by guesswork here :-))

It looked like the event loop integration worked (had prompt back, could pan/zoom).

As promised, I am surprised. "I wrote that code and I don't know how it's doing what you say it's doing" surprised ;-) Unless it was using the tk backend, as Python automatically integrates with the tkinter event loop without us needing to do anything.

If it does what you want and you don't need our completions or multiline input, maybe we should make that a more official feature rather than an internal testing flag.

Now that #9506 is "merged" I think we can get that in.

@takluyver, need some help to rebase ?

Rebased

Merged ! I'll open a small extra issue to improve deprecation warning and add them on import as well.