Removing outliers from a data set

[vivekgoyal]

Hi I have a set of values say about 1000 out of that i want to delete the value which are not in range, in a way removing outliers ..

I have tried calculating range, standard deviation but end up getting nothing

is there a quick way...


Thanks a lot for help...

 

[Hui]

Vivekgoyal


Firstly, Welcome to the Chandoo.org forums


A Typical methodology is to analyse your data and determine what level of Outliers you want to remove.

Lets assume you want to kepp data within +/-3 x Standard Deviations of the mean

A good way is to first Calculate the Mean and Standard Deviation of your Data

Work out the +3 and -3 SD values

Filter your data selecting values greater than +3 x SD

Delete them

Filter your data selecting values less than -3 x SD

Delete them


Voila


Of course that's just a 2 minute guide


There are a number of strategies for selecting data and identifying outlier values

The discussion of this is way too complex to attempt here.


That should be discussed with your peers/superiors who should be guiding you as to what that selection methodology is.

 

[vivekgoyal]

Thanks a lot Hui.