Large rework of how virtual text properties are stored, plus multiple other property bug fixes.#19685
Large rework of how virtual text properties are stored, plus multiple other property bug fixes.#19685paul-ollis wants to merge 9 commits intovim:masterfrom
Conversation
|
I wonder if this conflicts with #19661, that PR only modifies the drawing code. |
|
@64-bitman. Yes #19661 does conflict. I am clearly some way off getting a stable CI run, but assuming both PRs are accepted then I think there will be a tricky re-base regardless of which PR get merged first. |
h-east
left a comment
There was a problem hiding this comment.
About the document.
Write according to the rules.
:h help-writing
/^STYLEThe following two points are particularly important:
Vim help files generally use 2 spaces after a sentence (since they are written
using a fixed-width font and that was the preferred style in the 70s/80s),
like what is described here: https://english.stackexchange.com/a/2602
(snip...)
Use two spaces between the final dot of a sentence of the first letter of the
next sentence.
src/testdir/test_textprop.vim
Outdated
| edit foobar | ||
| set nowrap | ||
| set showbreak=+++\ | ||
| set showbreak=+++\ |
There was a problem hiding this comment.
This doesn't seem to be the intended fix.
There was a problem hiding this comment.
My vim configuration automatically removes trailing white space "noise" because, well it is "noise" and can only confuse.
I did notice the change had crept in but (a) the test still passes and (b) there is nothing to indicate that anyone thought the space to be important.
I see no good deliberately add the space back in.
There was a problem hiding this comment.
That's a bit of a stretch.
In Markdown files, removing trailing whitespace changes the layout.
Similarly, removing trailing whitespace in Vim scripts can cause unintended behavior or test failures.
Could you please revert it?
Below is a list (grep result) of lines that require trailing whitespace as intended for the test.
test_conceal.vim|151 col 24-26| set showbreak=\ >>>\
test_display.vim|345 col 21-23| set fillchars=eob:\
test_display.vim|416 col 28-30| set fillchars+=foldinner:\
test_listchars.vim|414 col 34-36| set listchars=tab:>-,leadtab:\|\
test_listchars.vim|422 col 35-37| set listchars=tab:>-,leadtab:│\
test_popupwin.vim|1168 col 19-21| set showbreak=>>\
test_scroll_opt.vim|520 col 49-51| set smoothscroll scrolloff=0 showbreak=+++\
test_signs.vim|168 col 42-44| sign define Sign5 linehl=Comment text=X\
test_textprop.vim|3609 col 24-26| set showbreak=+++\
test_textprop.vim|4276 col 20-22| set showbreak=>\
There was a problem hiding this comment.
The backslash in that value is only present to escape the trailing space character and include it in the value. The trailing backspace now makes no sense. Set it with :let if you like but the change is, I assume, irrelevant to the PR anyway.
It looks like the test passes because the wrapped line is now overwritten by the virtual text.
There was a problem hiding this comment.
@h-east and @dkearns. Thanks for your comments.
I have had a closer look at the test in question and the showbreak setting
appears to serve no purpose, except possibly to prove that showbreak has no
effect in the test's setup.
So, I have removed the "" character as well, which at prevents the test
remaining such a puzzle.
If anyone knows (or suspects) that the test is broken and should be investigated
further then please raise an issue.
There was a problem hiding this comment.
From 48ca24d it would appear it's included just to confirm that it's not drawn.
You're probably now aware but +++ is a pseudo default value as it's a prominent example at :help showbreak.
However, it's inconsistently used even in this test file and there are also some other examples of trailing single backslashes, which I've always considered a rather ugly exception to the \\ literal backslash rule.
:g/showbreak=
setlocal number showbreak=+ breakindent breakindentopt=shift:2
setlocal linebreak showbreak=+ breakindent breakindentopt=shift:2
call term_sendkeys(buf, ":set showbreak=+++\<CR>")
set showbreak=+++
set showbreak=+++
set showbreak=+++
set showbreak=+++\
call term_sendkeys(buf, ":set showbreak=>>\<CR>")
call term_sendkeys(buf, ":set showbreak=\<CR>")
set showbreak=>\
set cursorline scrolloff=0 showbreak=>\ smoothscroll
But let's not derail this commentary any further...
| buf_T *buf; // [R] The line's buffer. | ||
| linenr_T lnum; // [R] The number of the line. May be zero to | ||
| // indicate that no line is loaded. | ||
| bool detached; // [R] When set, "text" and property virtual text | ||
| // is allocated. | ||
| colnr_T text_size; // [R] Size of text including NUL. | ||
| char_u *text; // [R] Just the NUL terminated text. | ||
| uint16_t prop_size; // [P] The number of allocated "props". | ||
| uint16_t prop_count; // [R] The number of properties. | ||
| textprop_T *props; // [R] Variable length vector of properties. Treat | ||
| // as read-only unless detached is true. | ||
| bool text_changed; // [P] The text has been changed. Note "detached" | ||
| // must also be set. | ||
| int adjust_flags; // Flags controlling "prev" and "next" | ||
| // adjustments. | ||
|
|
||
| // Next and previous lines, used to adjust neighbouring properties. |
There was a problem hiding this comment.
Generally, single-sentence code comments do not contain periods. While there are no specific rules regarding this, I believe it's the same in Viim.
The same applies to code comments outside of this section.
There was a problem hiding this comment.
I try to respect the existing style of a code base but properly punctuated, single sentence comments appear
elsewhere within the code. I chose to follow that style because it fits with philosophy.
I much prefer properly punctuated comments, not least because it is respectful of the reader.
This simplifies the code in many ways as well as fixing a number of bugs and effective deliberate memory "leaking" required to make undo (mostly) work with properties. Signed-off-by: Paul Ollis <paul@cleversheep.org>
Moved text property testing support files to testdir/util.
- Moved text property testing support files to testdir/util. - Get memline more often in indent.c. May fix random lispindent failures. - Fix use of null pointer in textprop.c. - Added free for individual detached properties in charset.c. - Increased wait for a flaky screen dump test.
src/textprop.c
Outdated
| } | ||
|
|
||
| #endif // FEAT_PROP_POPUP | ||
|
|
There was a problem hiding this comment.
I think these notes should be at the top of the file
There was a problem hiding this comment.
@64-bitman. An interesting comment. I actually did have the notes at the top then moved the bottom because there was a lot of comment to scroll through in order to get to the code. In my (disturbingly) long career I have encountered more people who, in such cases, prefer such details to be tucked away at the bottom of the file.
I do not have strong feelings either way (at least as far as this PR goes) and will happily follow consensus or an authoritative steer from the core team.
There was a problem hiding this comment.
It seems we have a consensus of one, so I am moving the notes to the top of the file.
src/testdir/test_ins_complete.vim
Outdated
| " this was reading past the end of the line | ||
| new | ||
| norm 8oý | ||
| norm 8oý |
There was a problem hiding this comment.
This trailing space is inserted as buffer text. It should be kept.
There was a problem hiding this comment.
Extremely well spotted. I would never known.
Changing to execute "normal 8oý " so that this is no longer so fragile.
There was a problem hiding this comment.
@paul-ollis
Could you please revert the following settings?
My vim configuration automatically removes trailing white space "noise" because, well it is "noise" and can only confuse.
You are modifying parts that don't need to be changed, seemingly without realizing it. It's becoming counterproductive because you're trying to apply even more changes to address the unexpected side effects I've pointed out. These lines are there for a reason and are not "noise," so please leave them as they are.
ADD:
If changes are necessary, they should be made in a separate pull request, not this PR.
I would appreciate it if you would focus your efforts on the text property.
|
UPDATE: So, the CI build is stubbornly refusing to run clean because this error on "huge" builds. And I am frankly currently baffled. The problem does not occur for the "normal" build (the test script runs and passes fine). There should be no reason for the "huge" build to work as well. At the point of the failure, Vim is just sourcing some code and, as part of that, loading some Python 3 code. Python 3 is enabled and the Python 3 tests are running and passing. Any ideas, suggestions or help about how to figure out the problem would be very gratefully received. I have tried to replicate the "huge" build on my local PC and am pretty sure I have succeeded with something that is very close. For me the tests run fine. I am about to try some experimenting which will, temporarily, break other tests so I will make this PR draft for a while. |
In test_textprop2, i don't think you can use the import command in legacy vim script |
Thank for the suggestion, but - no it works. The test_textprop2.vim script works absolutely fine for the "normal" build. Once again thanks. |
... which I dumbly lost during a file move and branch cleanup.
chrisbra
left a comment
There was a problem hiding this comment.
I found a few issues. Some are style (like braces for single statements, using the FASE/TRUE macros for booleans, using free() instead of vim_free(). Then there is at least a potential double free in prop_add_one() and the error case in perform_properties_adjustment() on um_detach() failure looks a bit suspicious.
src/change.c
Outdated
| */ | ||
| void | ||
| del_lines(long nlines, int undo) | ||
| del_lines(long nlines,int undo, int flags) |
There was a problem hiding this comment.
please keep the space after nlines
There was a problem hiding this comment.
Fixed for next push.
| call StopVimInTerminal(buf) | ||
| endfunc | ||
|
|
||
| func Test_error_when_using_negative_id() |
There was a problem hiding this comment.
Why is this deleted? This test and the next one should be kept, no?
There was a problem hiding this comment.
Hi Chris, thanks for finding the time to review this.
Perhaps this is comment and the need to rebase the PR in light of #19741, is a
good trigger to discuss just totally simplifying positive and negative IDs.
[Please be aware that I was pretty exhausted by the time I submitted this PR.
What I expected to be a small fix turned into fixing non-trivial design flaws,
wading through technical debt and, frankly, some code that long ago became way
too hard to maintain.
By the end, the desire to keep the PR well focused conflicted with a desire to
simply get the PR submitted. I am painfully aware that I may have failed to
acheive the correct balance - for which I apologise. With that in mind...]
As I understand it, the restrictions on when negative IDs could be used were
only ever required because of the way that the text for virtual text properties
was separately stored; using negative IDs as an index. In other words, the
introduction of virtual text properties allowed implementation details to
leak into and complicate the text property API.
As a result of this PR the need for the restrictions disappeared and the
associated code covered by these tests "disolved", as I recall. The code for
using negative IDs for virtual text properties did "survive" albeit in
simplified form.
Thinking about this again, I cannot see any technical reason to place any
restrictions on IDs. That is, if the user supplies a value for the ID then
there is no technical reason that the provided ID should not be used.
The current (master) API rules feel like a bit of a mess to me. It would be
good to make them at least well defined before I rebase.
| textprop_T *tp = &text_props[used_tpi]; | ||
| char_u *p = tp->tp_id != -MAXCOL ? tp->tp_text : NULL; | ||
| int above = PROP_IS_ABOVE(tp); | ||
| int bail_out = FALSE; |
There was a problem hiding this comment.
I agree, but it is existing code. Is changing the types for "bail_out" and "above" acceptable code churn?
| #ifdef FEAT_PROP_POPUP | ||
| EXTERN char e_cannot_add_textprop_with_text_after_using_textprop_with_negative_id[] | ||
| INIT(= N_("E1339: Cannot add a textprop with text after using a textprop with a negative id")); | ||
| #endif |
There was a problem hiding this comment.
can you update the comment above?
| if (umemline->text != NULL) | ||
| { | ||
| umemline->text = vim_strsave(umemline->text); | ||
| } |
src/textprop.c
Outdated
| return TRUE; | ||
|
|
||
| fail: | ||
| free(umemline->text); |
src/textprop.c
Outdated
| for (int j = 0; j < i; j++) | ||
| { | ||
| free(umemline->props[j].tp_text); | ||
| } |
There was a problem hiding this comment.
I am wondering, if we should do um_abort() here?
src/textprop.c
Outdated
| int is_vtext = PROP_IS_VTEXT(prop); | ||
| prop_mod_T result = adjust(prop, col, count); | ||
| if (result != PC_UNMODIFIED && !um_detach(&umemline)) | ||
| break; |
There was a problem hiding this comment.
if um_detach() fails midway in this loop, we break out and then close the inconsistent umemline.
| buf->b_ml.ml_line_ptr = newtext; | ||
| buf->b_ml.ml_line_len += sizeof(textprop_T); | ||
| buf->b_ml.ml_flags |= ML_LINE_DIRTY; | ||
| tmp_prop.tp_text = virt_text; |
There was a problem hiding this comment.
this causes a double free in case of an error.
Co-authored-by: Christian Brabandt <cb@256bit.org>
h-east
left a comment
There was a problem hiding this comment.
I have strong reservations about addressing multiple distinct issues/improvements within a single PR. Whether such a major logic change is truly necessary remains a matter of debate. First and foremost, each issue should be handled individually.
Furthermore, I fail to understand the refusal to comply with requests from Vim members regarding non-essential parts of the PR.
#19685 (comment)
#19685 (comment)
|
Review result Bugs
Unused code The vtext_T struct (with ref_count) is defined in structs.h but never Code style
Tests Python test model (textprop.py, 1017 lines):
Removed tests:
Documentation
Unrelated changes The following are mixed into this PR but are unrelated to text property
Summary
|
|
I removed the Python dependency from test_textprop2.vim and made it a pure Vim script. (All 13 tests passed) textprop.py and textprop_support.vim widthin |
|
So I have been snatching odd, currently very precious, moments to try to keep This PR is big. It took up far more of my time than I expected, more than I It is quite likely that some major changes might be required for the PR, not But this is not a conversation and, as things stand, I am not willing to engage My previous contributions have been pleasant experiences and, I hope, useful. I can be contacted directly via email. |
|
@paul-ollis Participating in open source requires a degree of flexibility and a willingness to compromise. |
@paul-ollis Thank you for the significant effort you've put into this. The listener functionality is important for the Language Server Protocol support, and your work on improving this is appreciated. |
|
@paul-ollis thanks for all the work you put into Vim and improve the listener and text-properties support. Hope to see you back eventually |
My apologies for the size of this PR, but the fix for #19680 and #19681
required some major redesign/rework. During the course of those changes other
issues became apparent, which I felt could not be disentangled into separate
PRs.
The issues that this PR addresses are: #12568, #19256, #12677, #12678, #12679,
#12680, #12681, #12682, #12683 and #12684.
The major change is to store the text for virtual text properties within the
memlineinstead of a special table attached to the buffer. This change:Makes undo/redo robust when virtual text properties are used (Undo/redo can fail for virtual text properties. #19680).
Stops ever increasing memory usage by the table mentioned above (Virtual text properties consume ever increasing memory as they are added/deleted. #19681).
Removes the restriction about using negative property IDs.
The changes to the property help file reflects the removal of the above
restriction and also include some corrections/clarifications.
I created quite a few additional tests in
test_textprop2.vim, which usestextprop_support.vimandtextprop.pyto "model" properties. Thetextprop_support.vimprovides a "facade" class that delegates to a class intextprop.py.The reason for this structure and the use of Python is quite simply that I found
writing the "model" using VimScript too difficult. Perhaps another person might
be able turn the "facade" in
textprop_support.viminto a pure VimScriptimplementation. But, sorry, it will not be me.
There are also a number of small changes to test scripts, which allowed me to
run the tests reliably on my local PC.