Skip to content

Table export to CSV crashes for large tables with out-of-memory exception. #107

Closed

Description

It appears that the table export to CSV builds the export file content first in RAM before writing it to the specified file. If the table is very large the process is hitting the 2GB RAM limit of Windows 32bit applications. Then it crashes. No file gets written at all.

Activity

  1. rp- commented on Sep 19, 2014

    @rp-
    Contributor

    You are right, for streaming our sqltablemodel is not really useful as it will not free any unused data.
    But doing real streaming for import and export shouldn't be to hard to implement and was on my list anyway.

  2. rp- commented on Oct 3, 2014

    @rp-
    Contributor

    csv data is now streamed out and shouldn't fill up memory.
    50731cd

  3. justinclift commented on Oct 5, 2014

    @justinclift
    Member

    @beyonav Would you have time to test this out in the Windows nightly builds, and confirm back to us if it works?

  4. justinclift commented on Oct 6, 2014

    @justinclift
    Member

    Oops, I should have mentioned. 😄

    You can download the nightly windows builds from here:

        http://rp.oldsch00l.com/sqlitebrowser/sqlitebrowser.exe.xz

    .xz is a compressed file file, you can use 7-zip to uncompress it before installing.

  5. justinclift commented on Oct 6, 2014

    @justinclift
    Member

    Thanks for testing Jens. 😄 I'll close this issue here, because the original reported problem is fixed.

    With the quote characters, they're supposed to be optional. Testing here just now, in Export → "Table as CSV file" dialog there is a "Quote character" option. Change that from the default of ", to the third line down (no character). That seemed to work for me, although I don't have a lot of data in the example database I just tried it on. Does that work for you?

  6. justinclift commented on Oct 6, 2014

    @justinclift
    Member

    Is this the right spec? https://tools.ietf.org/html/rfc4180

    We had some discussion about CSV handing in #84. I guess we need to tweak things further still. 😄

  7. justinclift commented on Oct 6, 2014

    @justinclift
    Member

    Hmmm, we should probably turn this into a new Issue, for better tracking.

    Are you ok to do that? If not, just say so and I'll create one. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions