I have a CSV file that has data from a random sensor recorded over a few minutes time. Now I want to stream that data from a CSV file to my python code as if it were receiving data from the sensor itself directly. (The code is for taking readings from two different sensors/csv files and averaging them.) Someone suggested to use Apache Spark to stream data, but I feel that's a bit too complex for me. Might there be a simpler solution?
-
1Can't you just read it line by line? (and optional add a bit of delay between reading a line)Olian04– Olian042017-01-17 18:43:38 +00:00Commented Jan 17, 2017 at 18:43
-
1Are you wanting to mock the sensor interface or do you just care about the data? Do you need to have delays between reads? Is the data timestamped so you can calculate delays if you need them?Steven Rumbalski– Steven Rumbalski2017-01-17 18:44:39 +00:00Commented Jan 17, 2017 at 18:44
-
@Olian04 perhaps i could do that. i am pretty new to programming and have no idea about all this stuff. i apologize for sounding stupid. but will i have to give some time delay in between reading the lines to give it the real effect?M. Ali– M. Ali2017-01-18 15:23:27 +00:00Commented Jan 18, 2017 at 15:23
-
@M.Ali not necessarily. If the time between reading doesn't matter then you can just read it as you would any other file.Olian04– Olian042017-01-19 08:44:41 +00:00Commented Jan 19, 2017 at 8:44
-
There is an answer in stackoverflow.com/a/6556862/1548275Jari Turkia– Jari Turkia2021-11-12 10:46:27 +00:00Commented Nov 12, 2021 at 10:46
Add a comment
|
4 Answers
You could also use pandas read_csv() function to read the big csv file in small chunks, the basic code is written below:
import pandas as pd
chunksize = 100
for chunk in pd.read_csv('myfile.csv', chunksize=chunksize):
print(chunk)
This link explains how this works: http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking
Comments
you could use something like tail -f in python to achieve this. this should do what you want. http://lethain.com/tailing-in-python/
2 Comments
M. Ali
is it possible if I make a virtual serial device and transmit the data through that? i mean ultimately thats how i will interface the sensors anyway, so why not code according to that?
acushner
no idea, but probably? seems more complicated than just reading the csv files, unless you want to flip it around and have python interface directly with the sensors and have the python process also store csv data.
You can also work with Python on Numpy/Matplotlib. It´s a easy way to stream your csv data temporay as a variable and not a extra file.
´import matplotlib.pyplot as plt
from matplotlib import style
import numpy as np
import io
def draw_graph_stream(csv_content):
csv_stream = io.StringIO(csv_content)
svg_stream = io.StringIO()
data = np.genfromtxt(csv_stream, delimiter = ';') # generate the stream
x = data[0,:] #first row in csv
y = np.mean(data[1:,:], axis=0) # first column with mean generate the average
plt.plot(x,y)
plt.savefig(svg_stream, format = 'svg') #just safe it as svg
svg_stream.seek(0) #Position 0 for reading after writing
return svg_stream.read()
print("Start test")
with io.open('/filepathtodata','r') as csv_file: #works like a Loop
print("Reading file")
csv_content = csv_file.read()
print("Drawing graph")
svg_content = draw_graph_stream(csv_content)
with io.open('thefilepathforsafe','w+') as svg_file:
print("Write back")
svg_file.write(svg_content)´
Comments
It had worked in fastapi for streaming data to a user interface.
from starlette.responses import StreamingResponse
from io import BytesIO
temp_file_name = 'test.csv'
with open(temp_file_name, 'rb') as fh:
buffered_data = BytesIO(fh.read())
response = StreamingResponse(buffered_data, media_type="text/csv")
response.headers["Content-Disposition"] = f"attachment; filename=filename.csv"
return response
