Using Matrix-Analytic Strategies to Clean Noisy Data

Using Matrix-Analytic Strategies to Clean Noisy Data

CHALLENGE

The Internet of Things (IoT) refers to a connected network of computers, electronic sensors, and other technical components that collect, share, and analyze data for intelligent monitoring and automated decision making.  Similarly, the Industrial IoT applies these same network-of-sensor concepts to monitor processes in an industry setting in order to improve safety, efficiency, and profitability for the business organization.

A common problem in sensor applications involves noise-contamination of the time series signal that the instrument is tasked to measure. In many cases, the noise has straightforward properties for which it is possible to design simple statistical regression models, which can be used to smooth out the random perturbations and reveal the true signal that the sensor is measuring. However, if the noise magnitude is high relative to the power of the measured signal, or if the noise has complex statistical properties, then these strategies are ineffective.

For the case described here, a sensor was tasked to measure a two-dimensional image time series, in an environment where the noise was not only extremely powerful, but also exhibited widespread self-correlation properties. This is an unfortunate combination that can defeat even the most sophisticated noise rejection strategies if they are not carefully designed.


SOLUTION

Time series analysis strategies for removing noise in an image will typically involve convolution methods, whereby small sections are sequentially processed via a sliding window technique. However, for the case presented here, AlgoTactica designed a unique processing strategy involving a sequence of matrix-analytic operations, in which the entire image is processed as a whole matrix unit at each stage via canonical correlation methods. Using principles of feature engineering and stochastic process theory, the matrix numerical elements are strategically arranged to impose a special internal structure, which enhances the differences that the covariance properties of the noise exhibit in comparison to the properties of the underlying signal. It is this difference in covariance properties that forces the noise and the signal to separate into two separate matrix structures during the processing action, after which only the signal matrix is retained.


RESULTS

A comparison of the before and after images confirms that the technique is very effective in removing the noise contamination. In particular, the processed image reveals significant patterned regularities that could not be observed in the unprocessed time series image. Certain types of structure that might be found amongst these patterned regularities will provide useful information about the phenomenon that the sensor has been tasked to monitor; however, without noise removal it would not be possible to detect these.

The one-dimensional before & after time series plots from column 63 of each of the images further confirm the efficacy. These plots show that the resulting smoothed pattern (red) is not at all discernible in the unprocessed time series (grey), even though the unprocessed signal actually does contain this component.

-->
Back to top