Repository navigation
Improve RFC 4180 compliance of CSV import/export and add a quoted-as-text option - #4246
Open
deepsidhu1313 wants to merge 3 commits into
Open
deepsidhu1313 wants to merge 3 commits into
deepsidhu1313 wants to merge 3 commits into
Conversation
Import: - A line break which is a lone CR did not leave the end quote state when the last field of the line was quoted. If the first field of the next line was quoted, too, its opening quote was taken as an escaped quote and ended up in the field contents. - A last record consisting only of an empty quoted field was dropped when the file did not end with a line break. Export: - Fields containing a lone CR were not quoted when the line break was set to LF, so they were split into several records when read back. A lone LF was not quoted when the line break was set to something other than CR or LF. Values are now always quoted if they contain CR or LF. - Quoting was decided on UTF-8 bytes, so a field could be quoted because a byte of a multi-byte character was equal to the Latin-1 value of the quote character. - A column name containing a special character was written with NUL characters around it instead of quotes when no quote character was selected. The quoting rules now live in quoteCsvField() so they can be tested. The new tests cover embedded line breaks and quotes, the last record without line break, chunk boundaries and a round trip of exported fields through the parser. Related: sqlitebrowser#2445
The data type detection of the CSV import cannot tell a quoted value like "007" from an unquoted one, so a column of quoted numbers became an INTEGER column and lost its leading zeros. This is especially annoying when the CSV file comes from a tool which quotes text on purpose. The CSV parser now reports for each field whether it was enclosed in quote characters (CSVField::quoted). A new advanced option of the import dialog, "Treat quoted values as text", makes the import use this information: a column with quoted values gets the TEXT type when a new table is created, and quoted values are never converted to numbers when local number conventions are used. The option is off by default, so the import works as before unless the user turns it on. The new tests cover the quoted state reported by the parser and import a file through the dialog with and without the option. Related: sqlitebrowser#1382
With Qt5 a QTextStream uses the codec of the locale, which is not UTF-8 on Windows, so the UTF-8 test data was decoded wrongly there. Qt6 uses UTF-8 by default, like the other tests in this file already assume.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits.
Closes #2445, closes #1382