You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi! When I import a csv-file into a table every single space was deleted (v 3.3.0). Special characters are not displayed correctly even though in v 2.0 it worked fine. What happened? Is it my fault? Can I make any settings to avoid at least the space-deleting? Thanks a lot for your help!
Oh, that doesn't sound good. Do you have a part of the csv file you could share with us, so we can figure out what's going wrong? eg first few lines or something (they need to trigger the problem too 😄)
If you do, you're welcome to email it to me (justin@postgresql.org), and I'll forward it to Martin and Rene for investigating. 😄
Right now I can only give you 2 workarounds, until those bugs are fixed.
save your CSV file in the UTF-8 encoding, e..g with libreoffice calc (character encoding can be set on export)
Quote all your strings with whitespaces in it, I don't know if there is a CSV rfc or how strings in fields should really be handled, but i guess we shouldn't throw away whitespace between field separaters if they don't mess up the file format.
If the bug is something to do with us stripping whitespace around field separators, maybe we should add a checkbox to the import dialog about "preserve whitespace"?
The behaviour regarding spaces goes back to commit 37e195a - I can kind of see the point of that commit as well but, at the moment, would tend to remove lines 946-952. Any objections or better ideas? Maybe adding a checkbox for trimming the field contents would be a good compromise?
checkbox sounds reasonable, I would save the content of a field in a separate buffer to allow post processing as soon as we hit a new field.
It would also be nice to save the last used settings in the main config or per project.
Yes, feel free to do that 😃 Remembering the settings sounds good as well. I also though about extending the list in the encoding combobox automatically whenever a custom codec was used - e.g. user uses "ISO-8859-15", so it's safe to assume that this encoding might be used again. This maybe makes selecting encodings easier without having that endless list of never used formats in there.
Maybe the whole decode csv algorithm needs a rewrite, for example we don't support new lines in field data (if quoted), did this work with the old version?
And I too don't think the sqlitedb.cpp is a good place for this routine.
Right now I can only give you 2 workarounds, until those bugs are fixed.
save your CSV file in the UTF-8 encoding, e..g with libreoffice
calc (character encoding can be set on export)
Quote all your strings with whitespaces in it, I don't know if
there is a CSV rfc or how strings in fields should really be handled, but i
guess we shouldn't throw away whitespace between field separaters if they
don't mess up the file format.
—
Reply to this email directly or view it on GitHub #84 (comment)
.
Cool @Z4us, that's potentially useful. At the very least it gives us a minimum reference spec to follow. And it's only about 4 pages long of content. (unless other RFC's I've seen) 😄
@deichbilder This should now be fixed in the latest code. Would you be ok to try it, to confirm it works for you now? Not sure which operating system you're on... if it's Windows, there are nightly builds available here:
Great! I will try it asap (i'm on holidays for a few days). Os is Windows.
thanks for fixing it so quickly! I'm sure it will work. I'll give you
feedback.
Am 04.09.2014 17:57 schrieb "Justin Clift" notifications@github.com:
@deichbilderhttps://github.com/deichbilder This should now be fixed in
the latest code. Would you be ok to try it, to confirm it works for you
now? Not sure which operating system you're on... if it's Windows, there
are nightly builds available here:
I just tested - it works! :-) thanks a lot and have a nice sunday!
Am 04.09.2014 18:23 schrieb "Andrea Schmidt" a.schmidt.kollmar@gmail.com:
Great! I will try it asap (i'm on holidays for a few days). Os is Windows.
thanks for fixing it so quickly! I'm sure it will work. I'll give you
feedback.
Am 04.09.2014 17:57 schrieb "Justin Clift" notifications@github.com:
@deichbilderhttps://github.com/deichbilder This should now be fixed
in the latest code. Would you be ok to try it, to confirm it works for you
now? Not sure which operating system you're on... if it's Windows, there
are nightly builds available here:
Activity
justinclift commentedon Aug 28, 2014
Oh, that doesn't sound good. Do you have a part of the csv file you could share with us, so we can figure out what's going wrong? eg first few lines or something (they need to trigger the problem too 😄)
If you do, you're welcome to email it to me (justin@postgresql.org), and I'll forward it to Martin and Rene for investigating. 😄
rp- commentedon Aug 31, 2014
Right now I can only give you 2 workarounds, until those bugs are fixed.
rp- commentedon Aug 31, 2014
Fix list for us:
justinclift commentedon Aug 31, 2014
If the bug is something to do with us stripping whitespace around field separators, maybe we should add a checkbox to the import dialog about "preserve whitespace"?
rp- commentedon Aug 31, 2014
for some reason we don't add any spaces add all if they are not quoted see:
https://github.com/sqlitebrowser/sqlitebrowser/blob/master/src/sqlitedb.cpp#L948
and since we do line based parsing now, we should never get 0x10 or 0x13 characters (else if above)
MKleusberg commentedon Aug 31, 2014
The behaviour regarding spaces goes back to commit 37e195a - I can kind of see the point of that commit as well but, at the moment, would tend to remove lines 946-952. Any objections or better ideas? Maybe adding a checkbox for trimming the field contents would be a good compromise?
rp- commentedon Aug 31, 2014
checkbox sounds reasonable, I would save the content of a field in a separate buffer to allow post processing as soon as we hit a new field.
It would also be nice to save the last used settings in the main config or per project.
I can do that if you are busy Martin.
MKleusberg commentedon Aug 31, 2014
Yes, feel free to do that 😃 Remembering the settings sounds good as well. I also though about extending the list in the encoding combobox automatically whenever a custom codec was used - e.g. user uses "ISO-8859-15", so it's safe to assume that this encoding might be used again. This maybe makes selecting encodings easier without having that endless list of never used formats in there.
rp- commentedon Aug 31, 2014
Maybe the whole decode csv algorithm needs a rewrite, for example we don't support new lines in field data (if quoted), did this work with the old version?
And I too don't think the sqlitedb.cpp is a good place for this routine.
Z4us commentedon Sep 1, 2014
In fact there is: http://tools.ietf.org/html/rfc4180
As is written there spaces should not be ignored and use of double quotes
depends on system/application; M$ExCel goes it's own way...
Kind regards / Vriendelijke groeten / Cordiali saluti,
Klaas V
On Sun, Aug 31, 2014 at 8:38 AM, Peinthor Rene notifications@github.com
wrote:
justinclift commentedon Sep 1, 2014
Cool @Z4us, that's potentially useful. At the very least it gives us a minimum reference spec to follow. And it's only about 4 pages long of content. (unless other RFC's I've seen) 😄
justinclift commentedon Sep 4, 2014
@deichbilder This should now be fixed in the latest code. Would you be ok to try it, to confirm it works for you now? Not sure which operating system you're on... if it's Windows, there are nightly builds available here:
http://rp.oldsch00l.com/sqlitebrowser/sqlitebrowser.exe.xz
.xz is a type of compression. It can be uncompressed using 7-zip.
deichbilder commentedon Sep 4, 2014
Great! I will try it asap (i'm on holidays for a few days). Os is Windows.
thanks for fixing it so quickly! I'm sure it will work. I'll give you
feedback.
Am 04.09.2014 17:57 schrieb "Justin Clift" notifications@github.com:
deichbilder commentedon Sep 21, 2014
I just tested - it works! :-) thanks a lot and have a nice sunday!
Am 04.09.2014 18:23 schrieb "Andrea Schmidt" a.schmidt.kollmar@gmail.com:
justinclift commentedon Sep 22, 2014
Awesome, thanks for confirming. 😄