LSP Support in Vim Core #19621

For what it's worth, as mentioned by Yegappan, another solution is using his api diff() in this way to get the text diffs from a cached buffer. https://github.com/DanielViberg/lsp/blob/267785525e37e119cee7b3609872ce2052d34c91/lib/Features/DocumentSync.vim#L141-L165

In your opinion, should Vim implement an LSP client rather than being a component for LSP clients?

A C based LSP client built-in to vim?

It would be good to have. Is it feasible given the fact several independent plugins are available and the burden it would add for the maintainers? I don't know.

Thank you. We're currently in the phase of deciding which of those to pursue, so your input is very helpful.

I think this should be implemented in Vim9script so that more people can contribute to this. I looked into implementing the LSP support in core Vim a few years ago along the lines of the existing Netbeans support. But decided against doing that as only a few people will contribute to the C based implementation.

That would be that one here: #19003 I guess

I'm just curious, what makes implementing string identifiers non-trivial? Would it be possible to use a hash table to match LSP requests with responses?

@64-bitman Yes. The LSP feature depends on the Vim channel feature. The channel code can be extended to support string identifiers. Someone needs to work on this.

The issue is not that getbufline() itself is slow internally — it's that to compute a diff for incremental text synchronization in LSP, the Vim script LSP client must call getbufline(1, '$') to get the entire buffer content, even when only a single line has changed. For a 30,000-line buffer, that means copying 30,000 strings into a list just to find the one line that changed. The bottleneck is the unnecessary work of retrieving and diffing the full buffer, not the implementation of getbufline() per se. That's exactly why I proposed listener_add() or getchanges() — so the client can be notified of what changed without having to scan the entire buffer.

Short answer: yes, but the important part isn't "more detailed" — it's "self-contained".

Today's listener callback gives you only the coordinates of each change (lnum, end, added, col). To find out what text was actually there and what it became, you still have to call getbufline() yourself. The catch is that those coordinates are relative to the buffer state at the time the change fired, but getbufline() can only read the buffer state right now. If anything happens between the two — a later change in the same batch shifts lines, the callback is deferred, another plugin edits the buffer — the text you read no longer matches the range you were given.

This hurts especially when combined with debouncing, which LSP clients almost always want to do: sending didChange on every keystroke overwhelms the server, so changes are typically coalesced over a short window before being flushed. But the moment you defer reading, the coordinates you stored are no longer valid against the current buffer. You're forced to either (a) give up and send the full document, or (b) snapshot getbufline() inside every callback — which defeats the point of debouncing, since you've already paid the read cost up front.

A textdiffs / getchanges() style API would attach the actual before/after text to each change entry, so the callback delivers a complete, time-consistent record: "at tick N, range X was 'foo', now it's 'bar'". You never need to reconcile the coordinate with live buffer state, because the text was captured the moment the change happened. Reading and sending can be decoupled cleanly, and debouncing becomes trivial.

So:

• More detail? Yes — before/after strings instead of just line numbers.
• But the real value is removing the coordinate-vs-buffer-state race, which is what makes debounced incremental sync (LSP especially) fragile today. A line-level API would already be useful; character-level would be even better.

FWIW I'm actively working on a getchanges() patch for Vim core that does exactly this — it returns per-change before/after text captured at fire-time, so clients can debounce freely without losing the coordinate/text consistency.

@mattn @cagledw @yegappan May I ask in which case do you encounter failure or inconsistent result when using listener_add() to incrementally update lines of a buffer?

I encounter no errors using listener_add() for years, I do use listener_add() in roughly every keystrokes, in my completion-engine project which provide fast auto-completion and other things which need to keep a copy currently actively editing buffer.

If your apply diff logic is correct and its args feed to it via getbuflines() fetched in listener callback is also correct, then the result should not be wrong. The culprit might be something else, rather than listener_add()'s fault.

I swiftly skipped the related code I wrote, and do find a note telling that, timer_start is the real cause of failure/inconsistent result. If I remember correct, in the listener callback, textlock is lock, thus the buffer will not change when in the callback. But also you need to using timer_start to send to server the textchanged infos and related getbuflines() text.

Let's explain the culprit further, if two changes callback fired at a row, each one is correct, but with timer_start(), the second one could be send to server first, thus wrong result happens. The fix is simple, all the change needed is to create a list as queue to synchronize it, push to the queue in callback, pop from queue and send to server in the timer. After this change, the listener_add() will work all the time as long as the textchangd info is correct.

I believe it's better to first achieve the result in pure vimscript first, and find the common need and interface of all authors that need this capability. Straightly dive in to C do not save much, IMHO. The capability is fully achievable in current vim in pure Vimscript, change vim source promptly might introduce unnecessary bugs and has no noticeable performance gains.

I do agree the logic is a little complex, and there is no need to invent the wheel again and again, which also introduce countless subtle bugs, since it is quite hard to get it right.

I propose to take a different route, after agreeing on the external interface of this capability that most plugin authors agree upon, then provide a buffer#Watcher() function or something that do the work, probably provide it in vim itself, might in a pack, or a autoloaded function in vim itself so plugin developers can test it directly.

Later if the functionality is prove to be right, then it can be officially added to vim. If later do find that re-write in C can provide significant performance boost, then re-write it in C, otherwise several autoloaded functions & object will suffice.

@qeatzy
In general there is only one issue with listener_add() fcn that I'm aware of:
#19947

The rest of your post I had trouble following. FWIW, I suspect there are only a few of us looking vim LSP server/client support for vim (nvim much better - unfortunately). Apparently, some are satisfied with (https://github.com/prabirshrestha/vim-lsp), however I find that solution completely unusable for multiple reasons, starting with speed.

Additionally, I do have a work in progress, that isn't quite usable yet.
(https://github.com/cagledw/vim-oolsc)

@cagledw I suspect there is any issue with listener_add(), I'am using it since Bram era, and tested it extensively.
Has listener_add() had any correctness issue, then there will be also same issue for undo, which means enormous issues of bug report.

I just read your issue #19947, and be suspectable about it.
To verify/test, I tried again, using two buffers named from and to, edit in all crazy ways edit/paste/undo/etc in from and using listener_add() to mirror it.
Clearly no issue with vim, my vim version 9.1 Included patches: 1-265.

echo v:versionlong
9010265

@chrisbra @mattn @yegappan My concerns are without full understanding and testing, change the vim listener_add() source code will introduce bugs rather improve performance. I'm pretty sure as of #9768 listener_add()'s correctness is rock solid.
Concerning correctness of listener_add(), I do believe Bram's understanding of edit/undo, also used and tested listener_add() for years without any correctness issue.
To test correctness of listener_add() before change it (maybe already changed?), my proposal is create an official mirror fuction takes two buffers as argument, doing all crazy edits in buffer 1, and check if same contents in buffer 2, and let all user/plugin authors test it, any works should be after that.

I'am a auto-completion engine author which provide fast & fuzzy & automatic general completion, not a lsp writer yet, but my extensive usage/testing of listener_add() tells me it has no correctness issues since years ago.

FWIW, I suspect there are only a few of us looking vim LSP server/client support for vim (nvim much better - unfortunately).

@cagledw More proposal does not mean better code. The key is code quality & easy to use, Bram is good at former, but less good at latter. Bram do provide correct info with listener_add(), but correct interpretation of it is not provided, I'm ok to provide a solution for it, but apparently the listener_add() is already not longer same as Bram era, and do not provide correctness guarantee.

@chrisbra @mattn @yegappan
May I suggest, build it in two step.

• do not change listener_add() interface
• provide a apply change interface that guarantee correctness under all circumstances.
• provide the apply change interface in pure Vimscript, later port to C if it has noticeable performance gains.

rather than of mix the two and create a unreliable messy mixture.