0

I have a neural network that takes in arrays of 1024 doubles and returns a single value. I've trained it, and now I want to use it on a set of test data. The file that I want to test it on is very big (8.2G), and whenever I try to import it into Google Colab it crashes the RAM. I'm reading it in using read_csv as follows: testsignals = pd.read_csv("/content/drive/My Drive/MyFile.txt", delimiter="\n",header=None). What I'd like to know is if there's a more efficient way of reading in the file, or whether I will just have to work with smaller data sets.

Edit: following Prune's comment, I looked at the advice from the people who had commented, and I tried this:

import csv
testsignals=[]
with open('/content/drive/MyDrive/MyFile') as csvfile:
  reader=csv.reader(csvfile)
  for line in reader:
    testsignals.append(line)

But it still exceed the RAM.

If anyone can give me some help I'd be very grateful!

Improve this question
7
  • 1
    Please do the expected research before posting a question. The link is a starting point for a variety of solutions. Commented Aug 6, 2020 at 22:40
  • 2
    The basic problem is still the same: you're trying to read the entire file into a single data object, all at once. You do not have enough memory to hold it all. You have to read a chunk of data, process it, and dispose of it -- then you reuse the space for the next chunk. Look at the generator example. Commented Aug 7, 2020 at 0:06
  • 1
    in your code you append all lines to testsignals so it can't help you - you have to read few lines, use them and remove them from memory before you read few new lines. In pandas you can use for few_lines in read_csv(..., chunksize=rows_number): ... to read few lines. See read_csv Commented Aug 7, 2020 at 0:06
  • 1
    in pandas doc see also Iterating through files chunk by chunk Commented Aug 7, 2020 at 0:13
  • 1
    some models have the option of learning incrementally so you can feed it chunks of training data, scikit-learn has this implementation Commented Aug 7, 2020 at 0:22
Related questions
4
how to rapidaly load data into memory with python?
1
Pandas error while reading 14 GB csv file on 200 GB RAM workstation
3
How to resolve memory issue of pandas while reading big csv files
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.