Monitoring high-dimensional data for failure detection and localization in large-scale computing systems

doi:10.1109/TKDE.2007.190674

Monitoring high-dimensional data for failure detection and localization in large-scale computing systems

2008-01-01

23

PRE

AI

DOI：10.1109/TKDE.2007.190674

Original

Original request for help

Save

Abstract

En 中文

It is a major challenge to process high-dimensional measurements for failure detection and localization in large-scale computing systems. However, it is observed that in information systems, those measurements are usually located in a low-dimensional structure that is embedded in the high-dimensional space. From this perspective, a novel approach is proposed to model the geometry of underlying data generation and detect anomalies based on that model. We consider both linear and nonlinear data generation models. Two statistics, that is, the Hotelling T-2 and the squared prediction error ( SPE), are used to reflect data variations within and outside the model. We track the probabilistic density of extracted statistics to monitor the system's health. After a failure has been detected, a localization process is also proposed to find the most suspicious attributes related to the failure. Experimental results on both synthetic data and a real e-commerce application demonstrate the effectiveness of our approach in detecting and localizing failures in computing systems.

Keywords:

failure detection

manifold learning

statistics

data mining

information system

Internet applications

Monitoring high-dimensional data for failure detection and localization in large-scale computing systems

Abstract

AI Summary

Journal

Organization