Skip to content

import csv-file deletes spaces #84

Closed

Description

@deichbilder

Hi! When I import a csv-file into a table every single space was deleted (v 3.3.0). Special characters are not displayed correctly even though in v 2.0 it worked fine. What happened? Is it my fault? Can I make any settings to avoid at least the space-deleting? Thanks a lot for your help!

Activity

  1. justinclift commented on Aug 28, 2014

    @justinclift
    Member

    Oh, that doesn't sound good. Do you have a part of the csv file you could share with us, so we can figure out what's going wrong? eg first few lines or something (they need to trigger the problem too 😄)

    If you do, you're welcome to email it to me (justin@postgresql.org), and I'll forward it to Martin and Rene for investigating. 😄

  2. rp- commented on Aug 31, 2014

    @rp-
    Contributor

    Right now I can only give you 2 workarounds, until those bugs are fixed.

    1. save your CSV file in the UTF-8 encoding, e..g with libreoffice calc (character encoding can be set on export)
    2. Quote all your strings with whitespaces in it, I don't know if there is a CSV rfc or how strings in fields should really be handled, but i guess we shouldn't throw away whitespace between field separaters if they don't mess up the file format.
  3. rp- commented on Aug 31, 2014

    @rp-
    Contributor

    Fix list for us:

    • Add a text encoding combox in the import dialog, so the user can select the correct character encoding, I think right now we only support UTF8/Unicode
    • Fix the whitespace skipping
  4. justinclift commented on Aug 31, 2014

    @justinclift
    Member

    If the bug is something to do with us stripping whitespace around field separators, maybe we should add a checkbox to the import dialog about "preserve whitespace"?

  5. rp- commented on Aug 31, 2014

    @rp-
    Contributor

    for some reason we don't add any spaces add all if they are not quoted see:
    https://github.com/sqlitebrowser/sqlitebrowser/blob/master/src/sqlitedb.cpp#L948

    and since we do line based parsing now, we should never get 0x10 or 0x13 characters (else if above)

  6. MKleusberg commented on Aug 31, 2014

    @MKleusberg
    Member

    The behaviour regarding spaces goes back to commit 37e195a - I can kind of see the point of that commit as well but, at the moment, would tend to remove lines 946-952. Any objections or better ideas? Maybe adding a checkbox for trimming the field contents would be a good compromise?

  7. rp- commented on Aug 31, 2014

    @rp-
    Contributor

    checkbox sounds reasonable, I would save the content of a field in a separate buffer to allow post processing as soon as we hit a new field.
    It would also be nice to save the last used settings in the main config or per project.

    I can do that if you are busy Martin.

  8. MKleusberg commented on Aug 31, 2014

    @MKleusberg
    Member

    Yes, feel free to do that 😃 Remembering the settings sounds good as well. I also though about extending the list in the encoding combobox automatically whenever a custom codec was used - e.g. user uses "ISO-8859-15", so it's safe to assume that this encoding might be used again. This maybe makes selecting encodings easier without having that endless list of never used formats in there.

  9. rp- commented on Aug 31, 2014

    @rp-
    Contributor

    Maybe the whole decode csv algorithm needs a rewrite, for example we don't support new lines in field data (if quoted), did this work with the old version?
    And I too don't think the sqlitedb.cpp is a good place for this routine.

  10. Z4us commented on Sep 1, 2014

    @Z4us

    In fact there is: http://tools.ietf.org/html/rfc4180

    As is written there spaces should not be ignored and use of double quotes
    depends on system/application; M$ExCel goes it's own way...

    Kind regards / Vriendelijke groeten / Cordiali saluti,
    Klaas V

    On Sun, Aug 31, 2014 at 8:38 AM, Peinthor Rene notifications@github.com
    wrote:

    Right now I can only give you 2 workarounds, until those bugs are fixed.

    1. save your CSV file in the UTF-8 encoding, e..g with libreoffice
      calc (character encoding can be set on export)
    2. Quote all your strings with whitespaces in it, I don't know if
      there is a CSV rfc or how strings in fields should really be handled, but i
      guess we shouldn't throw away whitespace between field separaters if they
      don't mess up the file format.

    —
    Reply to this email directly or view it on GitHub
    #84 (comment)
    .

  11. justinclift commented on Sep 1, 2014

    @justinclift
    Member

    Cool @Z4us, that's potentially useful. At the very least it gives us a minimum reference spec to follow. And it's only about 4 pages long of content. (unless other RFC's I've seen) 😄

  12. justinclift commented on Sep 4, 2014

    @justinclift
    Member

    @deichbilder This should now be fixed in the latest code. Would you be ok to try it, to confirm it works for you now? Not sure which operating system you're on... if it's Windows, there are nightly builds available here:

        http://rp.oldsch00l.com/sqlitebrowser/sqlitebrowser.exe.xz

    .xz is a type of compression. It can be uncompressed using 7-zip.

  13. deichbilder commented on Sep 4, 2014

    @deichbilder
    Author

    Great! I will try it asap (i'm on holidays for a few days). Os is Windows.
    thanks for fixing it so quickly! I'm sure it will work. I'll give you
    feedback.
    Am 04.09.2014 17:57 schrieb "Justin Clift" notifications@github.com:

    @deichbilder https://github.com/deichbilder This should now be fixed in
    the latest code. Would you be ok to try it, to confirm it works for you
    now? Not sure which operating system you're on... if it's Windows, there
    are nightly builds available here:

    http://rp.oldsch00l.com/sqlitebrowser/sqlitebrowser.exe.xz
    

    .xz is a type of compression. It can be uncompressed using 7-zip
    http://www.7-zip.org/download.html.

    —
    Reply to this email directly or view it on GitHub
    #84 (comment)
    .

  14. deichbilder commented on Sep 21, 2014

    @deichbilder
    Author

    I just tested - it works! :-) thanks a lot and have a nice sunday!
    Am 04.09.2014 18:23 schrieb "Andrea Schmidt" a.schmidt.kollmar@gmail.com:

    Great! I will try it asap (i'm on holidays for a few days). Os is Windows.
    thanks for fixing it so quickly! I'm sure it will work. I'll give you
    feedback.
    Am 04.09.2014 17:57 schrieb "Justin Clift" notifications@github.com:

    @deichbilder https://github.com/deichbilder This should now be fixed
    in the latest code. Would you be ok to try it, to confirm it works for you
    now? Not sure which operating system you're on... if it's Windows, there
    are nightly builds available here:

    http://rp.oldsch00l.com/sqlitebrowser/sqlitebrowser.exe.xz
    

    .xz is a type of compression. It can be uncompressed using 7-zip
    http://www.7-zip.org/download.html.

    —
    Reply to this email directly or view it on GitHub
    #84 (comment)
    .

  15. justinclift commented on Sep 22, 2014

    @justinclift
    Member

    Awesome, thanks for confirming. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugConfirmed bugs or reports that are very likely to be bugs.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions