Skip to content

Cell edit window considers binary data to be text/numeric if first byte is in ascii range #1772

Closed

Description

@mbenningfield1

Details for the issue

What did you do?

Select a blob column while browsing a table.

What did you expect to see?

I expected to see the data represented as binary since the column affinity for the column is BLOB

What did you see instead?

On columns where the first byte of the binary header is outside of ASCII printable range, it shows the data as binary with the correct count of bytes.

The columns where the first byte of the binary header is within ASCII printable range, it shows the data as text/numeric with 1 char(s), instead of 32 bytes.

Useful extra information

The info below often helps, please fill it out if you're able to. :)

What operating system are you using?

  • Windows: ( _version:7 Pro version 6.1 Build 7601 SP1)

What is your DB4S version?

  • 3.11.1

Did you also

Activity

  1. added
    bugConfirmed bugs or reports that are very likely to be bugs.
    on Mar 1, 2019
  2. justinclift commented on Mar 1, 2019

    @justinclift
    Member

    Oh, interesting bug. At least, it sounds like a bug.

    Most of the time people seem to want things displayed as text if there's any reasonable chance of having it turn out ok. But yeah, for the situation where showing things in binary is preferred, that's been a lot less tested (AFAIK). Or at least, less issues have been opened about that approach.

    Hopefully we can get things worked out, so both use cases are catered to. 😄

  3. mbenningfield1 commented on Mar 1, 2019

    @mbenningfield1
    Author

    Yeah, I think trying to "massage" the representation in order to anticipate the user's expectation is doomed to failure. If the column has BLOB affinity, or the "Binary" tab in the cell edit window is active, just display the raw bytes. For example, a column with text or numeric affinity with a value of 1 will show a binary representation in the cell edit window (with the "Binary" tab active) as '0x31', which is binary for the ASCII character '1', not a hex value of 0x01. If I ask the sqlite engine for the same column value as a BLOB, it will give me one byte (unsigned char) as 0x01, NOT 0x031.

  4. mgrojo commented on Mar 1, 2019

    @mgrojo
    Member

    I'm not sure whether what you are seeing is it working as expected or a bug. In theory, If the first 512 bytes of a cell value have a correct UTF-8 encoding, they are displayed as text. But you say "The columns where the first byte of the binary header is within ASCII printable range, it shows the data as text/numeric with 1 char(s), instead of 32 bytes." I can't reproduce this, all the 32 bytes have to be printable in UTF-8.

    Could you post examples of BLOB detected as binary and BLOB detected as text? Screenshots might also help.

  5. sky5walk commented on Mar 1, 2019

    @sky5walk

    I do not follow mbenningfield1's claim?
    The Cell Editor has a dropdown for multiple formats.
    Choose Binary and the data is shown unaltered.
    In my case, I don't mind the Text view of my Blob cells since I often store plain text documents as Blobs.
    We had an issue regarding UTF-8 and ANSI showing different results in Text view, but you solved that already.

  6. mbenningfield1 commented on Mar 1, 2019

    @mbenningfield1
    Author

    The first shot is of a BLOB that shows up correctly, the second is a shot of one that does not.

    ok1
    notok1

  7. mbenningfield1 commented on Mar 1, 2019

    @mbenningfield1
    Author

    Sorry, I hit the wrong button and accidently closed this.

  8. sky5walk commented on Mar 1, 2019

    @sky5walk

    Again, confused. You are only showing the DB4S Binary view and claiming 1 is wrong.
    What is the actual contents of both cells? Can you show an external hex editor image?

  9. mbenningfield1 commented on Mar 1, 2019

    @mbenningfield1
    Author

    @sky5walk No, I'm not saying that the binary is showing the wrong bits. I'm saying that it isn't telling me the count of bytes for the binary, it it telling me there is 1 character in this BLOB. Working with binary data, I need the correct count of bytes.

  10. self-assigned this
    on Mar 1, 2019
  11. mgrojo commented on Mar 1, 2019

    @mgrojo
    Member

    I see now. There's a problem with the fix for #1731 when there are embedded nulls. The text detection is only working until the first null and the same for the character count.

  12. added this to the 3.11.2 milestone on Mar 2, 2019
  13. 5 remaining items

  14. mbenningfield1 commented on Mar 6, 2019

    @mbenningfield1
    Author

    I created a test table in my current database (id INTEGER, name TEXT, data BLOB) and imported the following lorem ipsum text file into the data column:
    LoremIpsum.txt

    On first touch, whether the Cell Edit window is in Text or Binary mode, the character count is correct. Switching from Binary to Text shows an incorrect lower count. Alternatively, clicking 'Auto-Detect' in Binary mode correctly detects as Text, but with the incorrect lower count. Alternatively again, if initially in Text mode, switching to Binary and back to Text results in the incorrect lower count.

    So, while my main concern has been addressed (correct byte count in Binary for binary data), it appears that the text detection process still needs a little work.

  15. justinclift commented on Mar 6, 2019

    @justinclift
    Member

    Thanks @mbenningfield1, that helps. 😄

  16. mgrojo commented on Mar 7, 2019

    @mgrojo
    Member

    I can reproduce the problem with the file that you've provided, but not with the ones I import from my local disk. It's clear that the problem is related to CR+LF endings of Windows. Qt seems to strip those CR at the minimal touch. If the file is exported again, it should be written in the local OS text format.

    https://forum.qt.io/topic/42464/qplaintextedit-eats-line-endings/5

  17. sky5walk commented on Mar 7, 2019

    @sky5walk

    Ok, good catch but super confusing and certainly a BUG.
    My column is a BLOB. Its contents cannot be stripped according to TEXT rules.
    My confusion was copy and paste from the cell edit window reapplied the [CR+LF] characters so I could not find the missing character count.

    Also, please change the nomenclature from Char(s) to Byte(s).
    Ansi and Unicode are a mess with Char(s).

  18. mgrojo commented on Mar 7, 2019

    @mgrojo
    Member

    My column is a BLOB. Its contents cannot be stripped according to TEXT rules.

    Well, you're editing it as text after all. Maybe the solution is to throw away the Qt text editor and use always the Scintilla version. I've seen that it is not string the CR characters, but at least in Linux, it is interpreting them as an additional end of line, leaving an empty line after every real line. How is it behaving in Windows? I've tested it using Wine and it is behaving as the 100% Linux version, which is strange to me.

    Also, please change the nomenclature from Char(s) to Byte(s).
    Ansi and Unicode are a mess with Char(s).

    I've always understood that as an abbreviation for characters, not bytes. In fact I changed it some time ago to count correctly characters for multibyte encodings. I saw that the count for Spanish was incorrect when the text included accented characters in UTF-8. We might want to change it to "character(s)" and maybe include also the byte count, but not sure about the available space.

  19. mgrojo commented on Mar 7, 2019

    @mgrojo
    Member

    I've updated "char(s)" to "character(s)". This is how most translators have understood it and the actual count. In the British "translation" I've also included the plural forms, so it will even read "1 character" or "n characters". This can only be made through a translation file, not in the original English text.

  20. mbenningfield1 commented on Mar 7, 2019

    @mbenningfield1
    Author

    @mgrojo: That sounds good. Should I close this issue and let someone open another one with the focus on text detection? As I said, my initial concern about binary data as BINARY appears to be resolved. I'm not that concerned about text detection, since I usually store large text files as TEXT. Is there anything else you guys need from me?

  21. mgrojo commented on Mar 7, 2019

    @mgrojo
    Member

    Thanks, @mbenningfield1. Yes, I think it's better to separate the issues.

  22. sky5walk commented on Mar 7, 2019

    @sky5walk

    My column is a BLOB. Its contents cannot be stripped according to TEXT rules.
    "mgrojo replied"
    Well, you're editing it as text after all. Maybe the solution is to throw away the Qt text editor and use always the Scintilla version. I've seen that it is not string the CR characters, but at least in Linux, it is interpreting them as an additional end of line, leaving an empty line after every real line. How is it behaving in Windows? I've tested it using Wine and it is behaving as the 100% Linux version, which is strange to me.

    Yes, I thought we discussed moving to the Scintilla gadget in another issue? You get the extra formatting/styling too!
    But regardless, there is still a bug if my BLOB content is altered while in the cell editor.
    Should I post an issue or can you reference one from here?

  23. justinclift commented on Mar 8, 2019

    @justinclift
    Member

    In the British "translation" ...

    Heh Heh Heh. From memory (years ago), that came from my needing something to play around with for gaining an understanding of Qt Linguist. We can probably drop it, unless someone really wants a proper UK spelling version and is willing to update it. 😄

  24. justinclift commented on Mar 8, 2019

    @justinclift
    Member

    But regardless, there is still a bug if my BLOB content is altered while in the cell editor.

    Yeah, sounds like a bug. Probably one that's a PITA to fix too. 🙄

    Should I post an issue or can you reference one from here?

    Post a new issue. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugConfirmed bugs or reports that are very likely to be bugs.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions