You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Cell Editor] Data detection / Qt edit gadget strip [CR's] from BLOB data. #1793
Create a BLOB column.
Populate with any multiline text.
View entry in Cell Editor and observe with the available options:
Text
Binary
Image
XML
JSON
What did you expect to see?
When selecting Binary, I expected to see the EXACT hex information I provided.
And the exact Byte count I provided.
What did you see instead?
The hex information shows my [CR] characters are stripped from the [CR+LF] line endings.
Only Hex 0A remains.
The DB4S Character(s) count is NOT accurate since the [CR]'s are missing.
Useful extra information
The Qt editor/DB4S code does reapply the [CR+LF] characters on export or cut and paste clipboard?
All very confusing.
The prior Issue mentioned dropping the Qt editor gadget and just use the Scintilla gadget for all modes.
Thanks @sky5walk. Have you confirmed that the Scintilla widget can load a Windows text file (a JSON or XML file with CR+LF endings) and behaves correctly in Windows, both in terms of displaying the lines and CR not being stripped?
Paste/Import json file contents with [CRLF] line endings.
Observe Character(s) calculated by DB4S.
Repeat 1-4 with other cell editor modes.
NOTE: Once the [CR] characters are stripped, the character count is invalidated.
Pasting with Binary cell mode corrupts the data?
I don't understand Binary?
INPUT -> JSON w/[CRLF] from Clipboard
CellMode
Character(s)
TEXT
FAIL <-- [CR]'s stripped
JSON
PASS
XML
PASS <-- Auto-detection reassigns mode to JSON.
BINARY
FAIL <-- Why is input data altered?
INPUT -> JSON w/[CRLF] from imported file.
CellMode
Character(s)
TEXT
FAIL <-- [CR]'s stripped
JSON
PASS
XML
PASS <-- Auto-detection reassigns mode to JSON.
BINARY
PASS <-- Auto-detection reassigns mode to JSON.
Conclusion:
Auto-detection should be disabled when user selects BINARY mode.
Scintilla gadget should be used for all non-binary views.
Yay 😄, the cell editor Scintilla gadget does not strip [CR]'s and I now have identical Character counts.
BUG(s):
If I hit the auto-detect wheel, the font is changed in the cell editor Scintilla gadget.
If I hit XML or any non-TEXT mode and return to TEXT, the font is changed.
It returns to my setting after closing and reopening DB4S.
The drawback is that pasting data from sources of different modes will convert the text in a mixture. This will probably be unusual when working in only one OS with the same DB.
If I hit the auto-detect wheel, the font is changed in the cell editor Scintilla gadget.
If I hit XML or any non-TEXT mode and return to TEXT, the font is changed.
It returns to my setting after closing and reopening DB4S.
I can't reproduce this. The text mode is supposed to use the same font as the Data Browser while the other modes use the same font as the SQL editor (a monotype font, more adequate for a data definition or programming language). This was already so, but now it is trickier because the widget is the same and the font has to switch. I had problems with this, but now is working good for me, so once again a platform issue?
You're right, the font used in the text mode is not the one configured for the data browser, as before, I think it's the font used by default by QScintilla. It is ignoring the setting. I'll have to make some testing or ask in the QScintilla mailing list.
@justinclift , Sorry, I tried many times but the nightly builds cannot show the text field any more in "Edit Data Cell" window, only if the language of the text is English, but not if text is in Arabic. This is a new issue that was not exist in the earlier releases. you can try the attached file, check the text field. sample.zip
You're right. I was thinking in using either the QChar::script() function (I think only Arabic, Syriac and Hebrew in that list are RTL scripts) or the QChar::direction() function (R, AL, RLE, RLO, RLI are RTL characters, according to Unicode).
It is another option. It would be simple, but then a user with a RTL layout could be using English or another language in the DB or viceversa, an English interface in the application and a DB in Arabic or Hebrew.
A little more digging found that functionality exists in QApplication. This would be detectable at startup in the main.cpp and could be set accordingly. The layout direction would then propagate downward to all created child widgets.
Until you decide which option is the right one , isn't it possible to develop 3.12 as a new branch to follow up with the new updates other than Qt widget. and keep the Qt widget .
Just a Question :)
Ok, almost done.
For existing database content, the binary view now shows the [CR][LF] characters.
But not when I paste new data into a blob field?
The [CR]'s are not deleted, but they are not shown in the binary view.
Activity
mgrojo commentedon Mar 8, 2019
Thanks @sky5walk. Have you confirmed that the Scintilla widget can load a Windows text file (a JSON or XML file with CR+LF endings) and behaves correctly in Windows, both in terms of displaying the lines and CR not being stripped?
sky5walk commentedon Mar 9, 2019
Yes, but with a simple test:
NOTE: Once the [CR] characters are stripped, the character count is invalidated.
Pasting with Binary cell mode corrupts the data?
I don't understand Binary?
INPUT -> JSON w/[CRLF] from Clipboard
INPUT -> JSON w/[CRLF] from imported file.
Conclusion:
Auto-detection should be disabled when user selects BINARY mode.
Scintilla gadget should be used for all non-binary views.
Drop the QTextEdit widget in the Edit Database Cell dock
sky5walk commentedon Mar 10, 2019
Yay 😄, the cell editor Scintilla gadget does not strip [CR]'s and I now have identical Character counts.
BUG(s):
If I hit the auto-detect wheel, the font is changed in the cell editor Scintilla gadget.
If I hit XML or any non-TEXT mode and return to TEXT, the font is changed.
It returns to my setting after closing and reopening DB4S.
Do we really need the auto-detect wheel❓
Johntaa commentedon Mar 10, 2019
Database cell editor doesn't accept LF & CR in text , I used to insert html entity
to substitute the LF. Hope it will be solved in the next release.
sky5walk commentedon Mar 10, 2019
Ooops, [LF]'s are converted to [CRLF]'s now. 😮
Still so much auto-detection??
Even if I paste into BLOB with Binary mode.
mgrojo commentedon Mar 10, 2019
If the only problem is when pasting, I think that using this Scintilla option will fix it:
https://www.scintilla.org/ScintillaDoc.html#SCI_GETPASTECONVERTENDINGS
The drawback is that pasting data from sources of different modes will convert the text in a mixture. This will probably be unusual when working in only one OS with the same DB.
I can't reproduce this. The text mode is supposed to use the same font as the Data Browser while the other modes use the same font as the SQL editor (a monotype font, more adequate for a data definition or programming language). This was already so, but now it is trickier because the widget is the same and the font has to switch. I had problems with this, but now is working good for me, so once again a platform issue?
sky5walk commentedon Mar 10, 2019
Yes, it seems to respect the content with File Import.
Only TEXT mode changes the font.
The other modes keep my original font.
mgrojo commentedon Mar 10, 2019
You're right, the font used in the text mode is not the one configured for the data browser, as before, I think it's the font used by default by QScintilla. It is ignoring the setting. I'll have to make some testing or ask in the QScintilla mailing list.
Restore correct font after the Cell Edit dock is set to plain text
mgrojo commentedon Mar 10, 2019
It should be fixed in tomorrow's nightly build.
sky5walk commentedon Mar 11, 2019
Nightly v190311 fixes the font change event. 👍
If @Johntaa is ok with the LF issue, (I do not have his use model), I would close this issue.
justinclift commentedon Mar 23, 2019
@Johntaa Would you have time to check the nightly builds, and see if the problem is fixed from your point of view? 😄
If you do, the nightly builds are here:
https://nightlies.sqlitebrowser.org/latest/
Johntaa commentedon Mar 24, 2019
@justinclift , Sorry, I tried many times but the nightly builds cannot show the text field any more in "Edit Data Cell" window, only if the language of the text is English, but not if text is in Arabic. This is a new issue that was not exist in the earlier releases. you can try the attached file, check the text field.
sample.zip
7 remaining items
scottfurry commentedon Jun 23, 2019
Google search: https://doc.qt.io/qt-5/qguiapplication.html#layoutDirection-prop
mgrojo commentedon Jun 23, 2019
You're right. I was thinking in using either the QChar::script() function (I think only Arabic, Syriac and Hebrew in that list are RTL scripts) or the QChar::direction() function (R, AL, RLE, RLO, RLI are RTL characters, according to Unicode).
mgrojo commentedon Jun 23, 2019
It is another option. It would be simple, but then a user with a RTL layout could be using English or another language in the DB or viceversa, an English interface in the application and a DB in Arabic or Hebrew.
scottfurry commentedon Jun 23, 2019
A little more digging found that functionality exists in QApplication. This would be detectable at startup in the main.cpp and could be set accordingly. The layout direction would then propagate downward to all created child widgets.
justinclift commentedon Jun 24, 2019
Sounds like it might be worth exploring. 😄
Johntaa commentedon Jun 25, 2019
Until you decide which option is the right one , isn't it possible to develop 3.12 as a new branch to follow up with the new updates other than Qt widget. and keep the Qt widget .
Just a Question :)
justinclift commentedon Jun 25, 2019
Oh yeah, there's a bunch of different possibilities for doing that. It just depends on who wants to take it up and how they want to do it. 😄
Restore QTextEdit widget for RTL text in the Cell Editor
New function for detecting mixed RTL-LTR texts
Restore QTextEdit widget for RTL text in the Cell Editor
New function for detecting mixed RTL-LTR texts
mgrojo commentedon Apr 26, 2020
@sky5walk Are all the problems address in 3.12 alpha?
sky5walk commentedon Apr 26, 2020
Ok, almost done.
For existing database content, the binary view now shows the [CR][LF] characters.
But not when I paste new data into a blob field?
The [CR]'s are not deleted, but they are not shown in the binary view.
justinclift commentedon Apr 27, 2020
As a general thought, I've been wondering if we should try building with the newest Qt release (5.14.2) too.
Apparently that has changes to better enable accessibility, which would really help with Community members needing to use screen readers and such.
Suspecting the newer Qt version might muck with some of our widgets (including the cell editor) again if we try though. 🤷