Learning Regularized Noise Contrastive Estimation for Robust Network Embedding

Hao Xiong

doi:10.1109/TKDE.2022.3148284

Learning Regularized Noise Contrastive Estimation for Robust Network Embedding

2022-01-01

1

OA

AI

DOI：10.1109/TKDE.2022.3148284

Original

Original request for help

Save

Abstract

En 中文

Skip-gram models are popular in large-scale network embedding for their cost-effectiveness. The objectives of many skip-gram based methods relate to the word2vec model which closely relates to Noise Contrastive Estimation (NCE). Among existing embedding methods, the differences mostly lie in how the node neighborhood is modeled e.g., by different ways of random walk, which leads to different learning strategies. Orthogonal to these efforts, we take a unified view that the NCE based methods commonly involve two basic NCE components in the learning objective. This perspective allows a natural generalization of the objectives by taking different forms of scoring function in the NCE components. We theoretically analyze how the vanilla NCE-based objectives suffer from the slow convergence speed and challenge in first-/second-order proximity preservation. We also prove the fundamental difficulty for NCE methods to capture non-linearity of complex networks. To mitigate such issues, we devise a general distance-based term added to the used NCE term, inspired by its physical meaning. The distance functions include Wasserstein-$k$k distance and Laplacian/Gaussian kernel functions, with relatively little additional time overhead. The effectiveness of our approach is verified both by prototype examples as well as real-world datasets, for the task of node classification and network reconstruction.

Keywords:

Estimation

Convergence

Linearity

Kernel

Task analysis

Predictive models

Data models

Network embedding

noise-contrastive estimation

score function

Learning Regularized Noise Contrastive Estimation for Robust Network Embedding

Abstract

Journal

Organization