How do I efficiently chart 200,000 data points

[wldm09]

Hello! This is my first post... so please be gentle...


I am running a datalogger that records 16 channels of temperature data on a locomotive. The measurements are logged 4 times per second. The logger is set to log for 4 hours after which it power cycles to create a new file. The files are easily transformed in to *.csv files. We download 6 files per day. On a high-output day, there are 50,000 rows in each file.


I would like to effeciently create line charts of this data and look at up to a weeks worth of data at a time. Help!


Thanks! wldm09

 

[richarta]

Graphing that many data points is unlikely to show anything of interest, except for a real mash of lines! It will be extremely slow in Excel, too.


So, you should aggregate the data - that is, reduce it to a smaller number of points by 'combining' values by taking their averages.


So, you have 50,000 rows. If you average every 10 values, the number of points is reduced to 5000. That is still too many, so you could average, say, in blocks of 100 or even 250. Alternatively, aggregate a previous aggregate - with, say, 10 values in each takes you down from 50,000 to 5000 to 500.


I must admit, I have not tried any of this in Excel, but it is the way some other tools work (namely, LoadRunner and BAC).


I hope this helps.


Tim