Support both kitty and sixel images - #5445
Conversation
|
Was looking for kitty support in tmux and saw this PR. Not sure if i'm doing something wrong, but getting: |
|
Oops sorry, forgot to add the new files. Added now. Please try again. |
Both kitty and sixel images seem to work. Plese let me know. There's also a rudimentary fallback to ascii in terminals with no image support at all. |
|
@senthil-instrumentl found another missing .h file which is now fixed (included inside image-kitty.h). Please give it a try. Feedback welcome. Thanks! |
|
I am using it inside Windows Terminal for development. I tried it briefly in kitty term. I fixed a flashing issue but that was already pushed yesterday. I can try it in wezterm probably tomorrow. Can you attach the png file or tell me where to get it? Hopefully we can fix the flashing everywhere. We will get this to work. Thanks for your help testing it! |
|
I Deleted my previous comments. Apologies for the email noise. My previous tests were invalid. However the PR sill does not work as expected for me. The images are rendered at incorrect position and do not move as the terminal scrolls. It also causes occasionally the statusbar to disappear when rendering or replaced by newline symbol. .tmux.conf has
Example: 
Using chafa produces similar results. I can not get kitty to work either. It will either do nothing or report unsupported. I am using WezTerm built from master couple days ago. All of these work flawlessly outside tmux. |
… image.c and the image-* backends.
Enable the sixel feature when WezTerm is detected and clarify that the kitty terminal feature requires Unicode placeholder support rather than only the basic graphics protocol.
|
I reproduced this in wezterm locally. I've added a fix for this to this branch, please pull it and recompile. IMPORTANT: you should remove this from your .tmux.conf Then fully restart the tmux server. tmux will detect each attached terminal separately:
If you have a separate reason to force sync, it should be configured only for the appropriate terminal pattern, not with *. The global line incorrectly claims every client supports both image protocols, which defeats per-client backend selection. |
A resize may temporarily leave the terminal cell pixel dimensions unknown while tmux queries the terminal. Schedule another client redraw when the pixel geometry response arrives so the graphical backend replaces the temporary text fallback.
|
I like the initiative, but there are a few bugs. diff --git a/image-sixel.c b/image-sixel.c
index 577ba51a..595c3836 100644
--- a/image-sixel.c
+++ b/image-sixel.c
@@ -429,9 +429,9 @@ sixel_colour_to_rgb(u_int colour, u_char *r, u_char *g, u_char *b)
double h, l, s, p, q;
if (type == 2) {
- *r = ((colour >> 16) & 0x1ff) * 255 / 100;
- *g = ((colour >> 8) & 0xff) * 255 / 100;
- *b = (colour & 0xff) * 255 / 100;
+ *r = (((colour >> 16) & 0xff) * 255 + 50) / 100;
+ *g = (((colour >> 8) & 0xff) * 255 + 50) / 100;
+ *b = ((colour & 0xff) * 255 + 50) / 100;
return;
}
if (type != 1) {
@@ -448,9 +448,9 @@ sixel_colour_to_rgb(u_int colour, u_char *r, u_char *g, u_char *b)
}
q = l < 0.5 ? l * (1 + s) : l + s - l * s;
p = 2 * l - q;
- *r = sixel_hue(p, q, h + 1.0 / 3) * 255;
- *g = sixel_hue(p, q, h) * 255;
- *b = sixel_hue(p, q, h - 1.0 / 3) * 255;
+ *r = sixel_hue(p, q, h) * 255 + 0.5;
+ *g = sixel_hue(p, q, h - 1.0 / 3) * 255 + 0.5;
+ *b = sixel_hue(p, q, h + 1.0 / 3) * 255 + 0.5;
}
/* Convert decoded SIXEL data into the protocol-neutral immutable image. */Second, sixel_from_image is... not pretty. The way you'd normally go about this is some kind of image quantization algorithm (I've had luck with octrees), with the current nearest neighbor approach the output looks awful, as demonstrated by your own screenshot. Or at the very least you should apply some error diffusion, ref. Third, sizing is broken. For example, should print a 26x26 image, but it gets scaled up to the cell size, so if I have 20x10 pixel cells then it blows up to 40x30. From a quick look at the code, Finally, maybe I'm wrong but it looks like the cropping Kitty parameters ( |
Use the DEC hue origin and channel order, and round percentage and HLS conversions to the nearest byte value.
Retain the padded canonical pixel canvas separately from the image content. Scale only the populated portion of partial edge cells so an image keeps its exact raster size on the originating terminal while retaining its cell footprint.
Build an adaptive 256-colour palette with median cut instead of using a fixed colour cube. Apply Floyd-Steinberg error diffusion while mapping pixels to reduce banding and preserve image detail.
@bptato, Did this address all your concerns? The sixel image definitely looks better to me now, thanks! |
Added.
done.
Ok shortened these up, also dimensions renamed to size. histogram to hgram in the struct, and hg in var, but left it histogram in the flags.
Exposed image backend_flags now.
relocated.
u_int64 was necessary in case of overflow which is highly unlikely. removed cast all together now, using width / xpixel + (width % xpixel != 0).
SIXEL and synchronized updates must now be enabled independently.
image cell drawing now moved into image.c |
|
OK, now Kitty seems to work right, but Sixel broke. (It's why I posted that link, but anyway...) #!/bin/sh
reset
printf '\033P0;1q"1;1;26;26#0;2;50;50;50#1;2;0;0;0#2;2;49;49;49#3;2;33;33;33#4;2;47;47;47#5;2;44;44;44#6;2;48;48;48#2?O?C!5?S!4?__?_?__$#0?_wW!5KGwo_-#0?~~!8?!10@BB}w$#2!12?a_A?A?A?A??@E-#0?~~!7?KK~~KK!7?~~$#2!9?K!6?K-#0?F^[owowowowowowowowooS^F$#2?G!10?@@!7?G??G-\033\'
tput cup 0 0
printf 'this line damages the first image\n'
printf '\n\n'
printf 'but on this line, the image will appear on top the text\r'
printf '\033P0;1q"1;1;26;26#0;2;50;50;50#1;2;0;0;0#2;2;49;49;49#3;2;33;33;33#4;2;47;47;47#5;2;44;44;44#6;2;48;48;48#2?O?C!5?S!4?__?_?__$#0?_wW!5KGwo_-#0?~~!8?!10@BB}w$#2!12?a_A?A?A?A??@E-#0?~~!7?KK~~KK!7?~~$#2!9?K!6?K-#0?F^[owowowowowowowowooS^F$#2?G!10?@@!7?G??G-\033\'
printf '\n\n' |
|
Here is what I see: |
|
wait, i think i see what you mean, the second line is right, the first line is wrong. |
…ne later overwritten by text. On a SIXEL terminal, damaged cells are omitted from image output; on Kitty they remain ordinary positional placements.
|
Try the latest push please. |
|
@bptato I was thinking about this last night, the current patch is not perfect, i may be able to do better. The issue is that tmux stores what the tty looks like internally so that when another client connects it can redraw it. tmux does not store the history of a cell (the temporal state of the cell). Storing a complete history of a cell (back to when it was empty) does not seem worth it. The current implementation does not store the state of overlapping sixel images. For example, if you try to composit 2 sixel images on top of one another, tmux would draw it, but if you reconnect or redraw the screen, the lower image would go away because it's no longer in the cell. The cell tracks which image (which part of which image) goes there, not a bit image of the cell. So what can I do? I can leave it as it is now which is once a cell has been overwritten by a character, any bit image in that cell is gone. This is definitely different than outside of tmux on ms term and probably other terminals. Or I can track a single image layer that basically tracks if there were bits written there before, don't clear first and just drop the text on top of it. That will make your test look correct but it won't work for more complex things like image1...image2...text where image2 has some transparent parts or does not fully cover image1 and would get composited on image1. Since we already store a single image ref and character for each cell, I am willing to store an order flag in the cell:
And if multiple images overlap, only the most recent image reference is retained for that cell. Any earlier image underneath is no longer represented there. This I think is quite easy to implement. I realise this isn't perfect but it seems like trying to preserve arbitrary z-ordering compositing of multiple images is going to add a ton of complexity for something which will never be used. What do you think? (edit) I have prepared a version that does the above. It wasn't super easy and it affected a bunch of files outside image-sixel.c. Please let me know how important is it to preserve this semantic even partially? My preference is the current behaviour because it is simple, but it is clearly not identical to other terminals. (edit 2) when there is both an image ref and a character in the cell, it was always drawing them both but first the character then the image. with what i just did, it preserves an order and more specifically does not clear the cell before ploping a character in it so that any remnant there remains there on purpose. It also fixed another bug. I am going to push this now, after some thinking, this really is the better of the two worlds, but it only preserves an image layer and text layer and which one was written first, not n-layers. if preserving n-layers is really important, let me know. |
…d first in the cell.
You don't have to store cell history. You "simply" have to store the images in the order they were received. (I know it's not simple...)
Yeah, at this point I'm skeptical of any approach that directly associates images with cells. At the very least, that association must be an ordered list, and that gets expensive (or complex) very fast. (Of course, it's a tradeoff, storing images in a separate list makes Sixel invalidation complex. Really the core issue is that Sixel was designed to be dumped onto the terminal's framebuffer, while Kitty intentionally decouples images from text, so combining them into a single model is difficult whatever you do.)
Well, it's broken, and I have no idea how the 'z' Kitty parameter can ever be implemented under these constraints (AFAICT right now it isn't?)
This makes no sense. Image/text mixing semantics are decided by the input format, not the output. The app sending the image has no idea about the outer terminal's output format. |
|
Perhaps cell history is not the proper way to describe it. Do I need to store each layer is really what I meant. By history, I mean that if you don't specify a z-index, the next image is above the previous one and if there are transparent bits you see through the one on top, down to older ones or ones lower in the z index. I can definitely do this and my question to you was is this necessary? and it sounds like from your throwing up your arms in disgust the answer sounds like yes. I was hoping to avoid doing this but if i have to i will. Your test cases have been super helpful. If you could prepare one or more that exercise the zindex more that would be helpful and I hope to get closer. I don't know how close I can get to working for you but hopefully I can get close enough.
Is this a comment again about the z-index? I thought by implementing this single pane I had more or less fixed this but it looks like I am going to have to go all the way and implement multiple and it's not the output dictating it, it's the input and z-index. Since sixel images do not support any sort of z-index placement, they are temporal, as in, the next one is on top of the previous. I have that info already, it's just a matter of wiring that to the cells. That should also make kitty z-index work. This all doesn't feel impossible, just a bit fiddly. Maybe I'm deluded? |
What do you want to achieve?
As for me, my main priority is, selfishly, that my TUI app does not work much worse after this patch than before. Admittedly, this doesn't affect many people, but it's a weaker version of 2, so I thought it'd be a useful direction for now.
No, it is a comment about how, according to the commit message, text & image interaction now depends on the outer terminal, and from a quick test, the commit message appears to be accurate. I'm saying this makes no sense, because the app that sent the image does not have information about the outer terminal. Hint: I think the image struct should have a "received from Sixel" flag that affects what happens after some text is printed over it. Edit: re-reading the conversation, maybe I wasn't clear enough about this: Sixel cannot be displayed behind text. If you print any text over a Sixel image, then the image disappears from the cells where you printed the text. So any logic of whether the image is "behind" the text is redundant; if it's behind the text, then it's already gone.
...by creating an array of images, then assigning an id to that array, and linking the array to the cell? |
Replace the per-cell image marker with sparse placement spans attached to grid lines. A placement owns all of its spans and records the input protocol, application image and placement IDs, z-index, and creation order. This retains overlapping image layers without storing a list in every grid cell. Grid operations move, split, clip, and remove only the affected spans. Use the input protocol to determine image/text interaction: later text damages SIXEL spans, while Kitty placements remain and are ordered by their z-index. Rendering then adapts that one logical scene for each client, rather than changing its semantics according to whether the outer terminal uses Kitty or SIXEL.
|
z-index hopefully now works. Thanks, your hint was right. The grid now retains every image placement as a separate ordered layer, represented sparsely as spans on the affected grid lines rather than as an array per cell. Each placement records its input protocol, IDs, z-index, and order. Previously, an image cell contained only one image reference. If image B overlapped image A, the cell could remember only B; tmux lost the fact that A should still be drawn underneath through B’s transparent pixels. Now the grid does not store an image list inside every cell. Instead:
On redraw, tmux finds the spans touching the area being rendered and orders them:
Within compatible layers, creation order breaks ties. This lets transparent overlapping images retain the layers below them, while avoiding an expensive per-cell linked list or array. Text writes now damage only SIXEL-origin spans. Kitty-origin placements retain their z semantics, including overlapping placements; the outer client’s Kitty/SIXEL capability no longer decides the logical image/text ordering. Hoping that makes sense now. The unavoidable output limitation is that SIXEL cannot draw an image beneath a text glyph. When rendering the same logical scene to SIXEL, tmux preserves text in those cells rather than emitting the underlying image portion. I installed Chawan to test tmux image scrolling and found that I needed -o buffer.images=true before inline images were fetched. Is keeping this opt-in intentional? Since it controls downloading images even when they cannot be displayed, I can see the bandwidth/privacy rationale; but would enabling it by default, or making it more discoverable, better match the expected browser experience? I tested the normal, non-passthrough path with Chawan on the WebP gallery, including scrolling, resizing, and Kitty/SIXEL clients attached to the same server. Could you please retry the original gallery-scrolling case? One separate Microsoft Terminal observation: its large SIXEL redraws need the independently advertised sync terminal feature to avoid visible intermediate repainting. This remains separate from SIXEL support; SIXEL does not imply Sync. Here is what I did in Windows Terminal which inhibits the flashing you might have seen: then in tmux: It scrolls smoothly directly in Windows Terminal and Kitty (outside of tmux). I also tested this in Wezterm. Albeit with this single web gallery page. How close are we to getting this working? Please do let me know if you find some issues. |
That sounds like a correct solution, thanks.
Yes, it's horrible UX. I wish it was enabled by default too, but I couldn't do that for a multitude of reasons - some still relevant, others less so.
Indeed, it seems to be working correctly now. Thank you!
I also see it with xterm (which has no Sync), it's what I alluded to with the "performance characteristics" comment before. |
|
Oh great, glad it works finally! |
5 participants
This PR adds generalised support for both kitty and sixel images. Basically it breaks up images into grid size rectangles. If the terminal is kitty then it sends kitty the image using the special unicode character encoding and kitty displays it locally. If the terminal supports sixel images, the image is sent to the terminal using sixel escape codes. This uses the new screen-redraw scene and span drawing, so it works in and around floating panes. Images can be scrolled into the copy buffer without issue.