Ive J asked . 2022-05-04

How to detect noisy data / outliers?

I have a problem to distinguish between a complete noisy data or data containing some outliers. In fact, I am trying to do a sort of Grid Independence Check (term usually is used in Computational Fluid Dynamics to identify the greed size where the numerical results are independent from the grid/network size) for my data set. I introduced this term only to better explain my case: I have a model which outputs a set of numerical data. Changing the step size in the model, increases the size of output numerical result (obviously). However, the output results contain noisy data which should be removed prior to the check for grid independence (or step size independence). An example output from the model can be similar to the following image.

rmvIdx = (abs(DataPoints- median(DataPoints)) > N*mad(DataPoints));
DataPoints(rmvIdx) = [];

Nevertheless, both the approaches miss some data points, and cannot "completely" detect outliers. Therefore, my first question is: how to fully detect outliers in the depicted image. I followed several variations to fully detect the outliers, e.g.:

% First remove distant outliers with N = 3
 rmvIdx = (abs(DataPoints- median(DataPoints)) > N*mad(DataPoints));
 DataPoints(rmvIdx) = [];

% Remove using local SD
Local_std = stdfilt(DataPoints);
Local_std(abs(Local_std- mode(Local_std)) < 1e-4) = 0;
DataPoints(Local_std) = [];

% Smoothing data as a further step
DataPoints= smooth(DataPoints,0.6);

AI, Data Science, and Statistics , Statistics and Machine Learning Toolbox , noise detection , outl