You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cell edit window considers binary data to be text/numeric if first byte is in ascii range #1772
I expected to see the data represented as binary since the column affinity for the column is BLOB
What did you see instead?
On columns where the first byte of the binary header is outside of ASCII printable range, it shows the data as binary with the correct count of bytes.
The columns where the first byte of the binary header is within ASCII printable range, it shows the data as text/numeric with 1 char(s), instead of 32 bytes.
Useful extra information
The info below often helps, please fill it out if you're able to. :)
What operating system are you using?
Windows: ( _version:7 Pro version 6.1 Build 7601 SP1)
Oh, interesting bug. At least, it sounds like a bug.
Most of the time people seem to want things displayed as text if there's any reasonable chance of having it turn out ok. But yeah, for the situation where showing things in binary is preferred, that's been a lot less tested (AFAIK). Or at least, less issues have been opened about that approach.
Hopefully we can get things worked out, so both use cases are catered to. 😄
Yeah, I think trying to "massage" the representation in order to anticipate the user's expectation is doomed to failure. If the column has BLOB affinity, or the "Binary" tab in the cell edit window is active, just display the raw bytes. For example, a column with text or numeric affinity with a value of 1 will show a binary representation in the cell edit window (with the "Binary" tab active) as '0x31', which is binary for the ASCII character '1', not a hex value of 0x01. If I ask the sqlite engine for the same column value as a BLOB, it will give me one byte (unsigned char) as 0x01, NOT 0x031.
I'm not sure whether what you are seeing is it working as expected or a bug. In theory, If the first 512 bytes of a cell value have a correct UTF-8 encoding, they are displayed as text. But you say "The columns where the first byte of the binary header is within ASCII printable range, it shows the data as text/numeric with 1 char(s), instead of 32 bytes." I can't reproduce this, all the 32 bytes have to be printable in UTF-8.
Could you post examples of BLOB detected as binary and BLOB detected as text? Screenshots might also help.
I do not follow mbenningfield1's claim?
The Cell Editor has a dropdown for multiple formats.
Choose Binary and the data is shown unaltered.
In my case, I don't mind the Text view of my Blob cells since I often store plain text documents as Blobs.
We had an issue regarding UTF-8 and ANSI showing different results in Text view, but you solved that already.
Again, confused. You are only showing the DB4S Binary view and claiming 1 is wrong.
What is the actual contents of both cells? Can you show an external hex editor image?
@sky5walk No, I'm not saying that the binary is showing the wrong bits. I'm saying that it isn't telling me the count of bytes for the binary, it it telling me there is 1 character in this BLOB. Working with binary data, I need the correct count of bytes.
I see now. There's a problem with the fix for #1731 when there are embedded nulls. The text detection is only working until the first null and the same for the character count.
I created a test table in my current database (id INTEGER, name TEXT, data BLOB) and imported the following lorem ipsum text file into the data column: LoremIpsum.txt
On first touch, whether the Cell Edit window is in Text or Binary mode, the character count is correct. Switching from Binary to Text shows an incorrect lower count. Alternatively, clicking 'Auto-Detect' in Binary mode correctly detects as Text, but with the incorrect lower count. Alternatively again, if initially in Text mode, switching to Binary and back to Text results in the incorrect lower count.
So, while my main concern has been addressed (correct byte count in Binary for binary data), it appears that the text detection process still needs a little work.
I can reproduce the problem with the file that you've provided, but not with the ones I import from my local disk. It's clear that the problem is related to CR+LF endings of Windows. Qt seems to strip those CR at the minimal touch. If the file is exported again, it should be written in the local OS text format.
Ok, good catch but super confusing and certainly a BUG.
My column is a BLOB. Its contents cannot be stripped according to TEXT rules.
My confusion was copy and paste from the cell edit window reapplied the [CR+LF] characters so I could not find the missing character count.
Also, please change the nomenclature from Char(s) to Byte(s).
Ansi and Unicode are a mess with Char(s).
My column is a BLOB. Its contents cannot be stripped according to TEXT rules.
Well, you're editing it as text after all. Maybe the solution is to throw away the Qt text editor and use always the Scintilla version. I've seen that it is not string the CR characters, but at least in Linux, it is interpreting them as an additional end of line, leaving an empty line after every real line. How is it behaving in Windows? I've tested it using Wine and it is behaving as the 100% Linux version, which is strange to me.
Also, please change the nomenclature from Char(s) to Byte(s).
Ansi and Unicode are a mess with Char(s).
I've always understood that as an abbreviation for characters, not bytes. In fact I changed it some time ago to count correctly characters for multibyte encodings. I saw that the count for Spanish was incorrect when the text included accented characters in UTF-8. We might want to change it to "character(s)" and maybe include also the byte count, but not sure about the available space.
I've updated "char(s)" to "character(s)". This is how most translators have understood it and the actual count. In the British "translation" I've also included the plural forms, so it will even read "1 character" or "n characters". This can only be made through a translation file, not in the original English text.
@mgrojo: That sounds good. Should I close this issue and let someone open another one with the focus on text detection? As I said, my initial concern about binary data as BINARY appears to be resolved. I'm not that concerned about text detection, since I usually store large text files as TEXT. Is there anything else you guys need from me?
My column is a BLOB. Its contents cannot be stripped according to TEXT rules.
"mgrojo replied"
Well, you're editing it as text after all. Maybe the solution is to throw away the Qt text editor and use always the Scintilla version. I've seen that it is not string the CR characters, but at least in Linux, it is interpreting them as an additional end of line, leaving an empty line after every real line. How is it behaving in Windows? I've tested it using Wine and it is behaving as the 100% Linux version, which is strange to me.
Yes, I thought we discussed moving to the Scintilla gadget in another issue? You get the extra formatting/styling too!
But regardless, there is still a bug if my BLOB content is altered while in the cell editor.
Should I post an issue or can you reference one from here?
Heh Heh Heh. From memory (years ago), that came from my needing something to play around with for gaining an understanding of Qt Linguist. We can probably drop it, unless someone really wants a proper UK spelling version and is willing to update it. 😄
Activity
justinclift commentedon Mar 1, 2019
Oh, interesting bug. At least, it sounds like a bug.
Most of the time people seem to want things displayed as text if there's any reasonable chance of having it turn out ok. But yeah, for the situation where showing things in binary is preferred, that's been a lot less tested (AFAIK). Or at least, less issues have been opened about that approach.
Hopefully we can get things worked out, so both use cases are catered to. 😄
mbenningfield1 commentedon Mar 1, 2019
Yeah, I think trying to "massage" the representation in order to anticipate the user's expectation is doomed to failure. If the column has BLOB affinity, or the "Binary" tab in the cell edit window is active, just display the raw bytes. For example, a column with text or numeric affinity with a value of 1 will show a binary representation in the cell edit window (with the "Binary" tab active) as '0x31', which is binary for the ASCII character '1', not a hex value of 0x01. If I ask the sqlite engine for the same column value as a BLOB, it will give me one byte (unsigned char) as 0x01, NOT 0x031.
mgrojo commentedon Mar 1, 2019
I'm not sure whether what you are seeing is it working as expected or a bug. In theory, If the first 512 bytes of a cell value have a correct UTF-8 encoding, they are displayed as text. But you say "The columns where the first byte of the binary header is within ASCII printable range, it shows the data as text/numeric with 1 char(s), instead of 32 bytes." I can't reproduce this, all the 32 bytes have to be printable in UTF-8.
Could you post examples of BLOB detected as binary and BLOB detected as text? Screenshots might also help.
sky5walk commentedon Mar 1, 2019
I do not follow mbenningfield1's claim?
The Cell Editor has a dropdown for multiple formats.
Choose Binary and the data is shown unaltered.
In my case, I don't mind the Text view of my Blob cells since I often store plain text documents as Blobs.
We had an issue regarding UTF-8 and ANSI showing different results in Text view, but you solved that already.
mbenningfield1 commentedon Mar 1, 2019
The first shot is of a BLOB that shows up correctly, the second is a shot of one that does not.
mbenningfield1 commentedon Mar 1, 2019
Sorry, I hit the wrong button and accidently closed this.
sky5walk commentedon Mar 1, 2019
Again, confused. You are only showing the DB4S Binary view and claiming 1 is wrong.
What is the actual contents of both cells? Can you show an external hex editor image?
mbenningfield1 commentedon Mar 1, 2019
@sky5walk No, I'm not saying that the binary is showing the wrong bits. I'm saying that it isn't telling me the count of bytes for the binary, it it telling me there is 1 character in this BLOB. Working with binary data, I need the correct count of bytes.
mgrojo commentedon Mar 1, 2019
I see now. There's a problem with the fix for #1731 when there are embedded nulls. The text detection is only working until the first null and the same for the character count.
Fix text detection for blobs containing bytes in ASCII range up to a …
5 remaining items
mbenningfield1 commentedon Mar 6, 2019
I created a test table in my current database (id INTEGER, name TEXT, data BLOB) and imported the following lorem ipsum text file into the data column:
LoremIpsum.txt
On first touch, whether the Cell Edit window is in Text or Binary mode, the character count is correct. Switching from Binary to Text shows an incorrect lower count. Alternatively, clicking 'Auto-Detect' in Binary mode correctly detects as Text, but with the incorrect lower count. Alternatively again, if initially in Text mode, switching to Binary and back to Text results in the incorrect lower count.
So, while my main concern has been addressed (correct byte count in Binary for binary data), it appears that the text detection process still needs a little work.
justinclift commentedon Mar 6, 2019
Thanks @mbenningfield1, that helps. 😄
mgrojo commentedon Mar 7, 2019
I can reproduce the problem with the file that you've provided, but not with the ones I import from my local disk. It's clear that the problem is related to CR+LF endings of Windows. Qt seems to strip those CR at the minimal touch. If the file is exported again, it should be written in the local OS text format.
https://forum.qt.io/topic/42464/qplaintextedit-eats-line-endings/5
sky5walk commentedon Mar 7, 2019
Ok, good catch but super confusing and certainly a BUG.
My column is a BLOB. Its contents cannot be stripped according to TEXT rules.
My confusion was copy and paste from the cell edit window reapplied the [CR+LF] characters so I could not find the missing character count.
Also, please change the nomenclature from Char(s) to Byte(s).
Ansi and Unicode are a mess with Char(s).
mgrojo commentedon Mar 7, 2019
Well, you're editing it as text after all. Maybe the solution is to throw away the Qt text editor and use always the Scintilla version. I've seen that it is not string the CR characters, but at least in Linux, it is interpreting them as an additional end of line, leaving an empty line after every real line. How is it behaving in Windows? I've tested it using Wine and it is behaving as the 100% Linux version, which is strange to me.
I've always understood that as an abbreviation for characters, not bytes. In fact I changed it some time ago to count correctly characters for multibyte encodings. I saw that the count for Spanish was incorrect when the text included accented characters in UTF-8. We might want to change it to "character(s)" and maybe include also the byte count, but not sure about the available space.
Make clear we are counting characters and not bytes or C char(s)
mgrojo commentedon Mar 7, 2019
I've updated "char(s)" to "character(s)". This is how most translators have understood it and the actual count. In the British "translation" I've also included the plural forms, so it will even read "1 character" or "n characters". This can only be made through a translation file, not in the original English text.
mbenningfield1 commentedon Mar 7, 2019
@mgrojo: That sounds good. Should I close this issue and let someone open another one with the focus on text detection? As I said, my initial concern about binary data as BINARY appears to be resolved. I'm not that concerned about text detection, since I usually store large text files as TEXT. Is there anything else you guys need from me?
mgrojo commentedon Mar 7, 2019
Thanks, @mbenningfield1. Yes, I think it's better to separate the issues.
sky5walk commentedon Mar 7, 2019
Yes, I thought we discussed moving to the Scintilla gadget in another issue? You get the extra formatting/styling too!
But regardless, there is still a bug if my BLOB content is altered while in the cell editor.
Should I post an issue or can you reference one from here?
justinclift commentedon Mar 8, 2019
Heh Heh Heh. From memory (years ago), that came from my needing something to play around with for gaining an understanding of Qt Linguist. We can probably drop it, unless someone really wants a proper UK spelling version and is willing to update it. 😄
justinclift commentedon Mar 8, 2019
Yeah, sounds like a bug. Probably one that's a PITA to fix too. 🙄
Post a new issue. 😄
Fix text detection for blobs containing bytes in ASCII range up to a …
Detect data with embedded zeros as binary also for ISO-8859