FORCE: Fast Outlier-Robust Correlation Estimation via Streaming Quantile Approximation for High-Dimensional Data Streams

Sooyoung Jang

doi:10.3390/math14010191

FORCE: Fast Outlier-Robust Correlation Estimation via Streaming Quantile Approximation for High-Dimensional Data Streams

2026-01-04

1

OA

AI

DOI：10.3390/math14010191

原文链接

原文求助

摘要

En 中文

The estimation of correlation matrices in high-dimensional data streams presents a fundamental conflict between computational efficiency and statistical robustness. Moment-based estimators, such as Pearson's correlation, offer linear O(N) complexity but lack robustness. In contrast, high-breakdown methods like the minimum covariance determinant (MCD) are computationally prohibitive (O(Np2+p3)) for real-time applications. This paper introduces Fast Outlier-Robust Correlation Estimation (FORCE), a streaming algorithm that performs adaptive coordinate-wise trimming using the P2 algorithm for streaming quantile approximation, requiring only O(p) memory independent of stream length. We evaluate FORCE against six baseline algorithms-including exact trimmed methods (TP-Exact, TP-TER) that use O(NlogN) sorting with O(Np) storage-across five benchmark datasets spanning synthetic, financial, medical, and genomic domains. FORCE achieves speedups of approximately 470x over FastMCD and 3.9x over Spearman's rank correlation. On S&P 500 financial data, coordinate-wise trimmed methods substantially outperform FastMCD: TP-Exact achieves the best RMSE (0.0902), followed by TP-TER (0.0909) and FORCE (0.1186), compared to FastMCD's 0.1606. This result demonstrates that coordinate-wise trimming better accommodates volatility clustering in financial time series than multivariate outlier exclusion. FORCE achieves 76% of TP-Exact's accuracy while requiring 104x less memory, enabling robust estimation in true streaming environments where data cannot be retained for batch processing. We validate the 25% breakdown point shared by all IQR-based trimmed methods using the ODDS-satellite benchmark (31.7% contamination), confirming identical degradation for FORCE, TP-Exact, and TP-TER. For memory-constrained streaming applications with contamination below 25%, FORCE provides the only viable path to robust correlation estimation with bounded memory.

Keyword:

robust statistics

correlation estimation

machine learning

streaming algorithms

quantile approximation

high-dimensional data

outlier detection

computational efficiency

breakdown point

memory-efficient computing

FORCE: Fast Outlier-Robust Correlation Estimation via Streaming Quantile Approximation for High-Dimensional Data Streams

摘要

期刊

机构

引用论文