Boosted negative sampling by quadratically constrained entropy maximization

Taygun Kekeç

doi:10.1016/j.patrec.2019.04.027

Boosted negative sampling by quadratically constrained entropy maximization

2019-07-01

2

PRE

AI

DOI：10.1016/j.patrec.2019.04.027

原文链接

原文求助

摘要

En 中文

Learning probability densities for natural language representations is a difficult problem because language is inherently sparse and high-dimensional. Negative sampling is a popular and effective way to avoid intractable maximum likelihood problems, but it requires correct specification of the sampling distribution. Previous state of the art methods rely on heuristic distributions that appear to do well in practice. In this work, we define conditions for optimal sampling distributions and demonstrate how to approximate them using Quadratically Constrained Entropy Maximization (QCEM). Our analysis shows that state of the art heuristics are restrictive approximations to our proposed framework. To demonstrate the merits of our formulation, we apply QCEM to matching synthetic exponential family distributions and to finding high dimensional word embedding vectors for English. We are able to achieve faster inference on synthetic experiments and improve the correlation on semantic similarity evaluations on the Rare Words dataset by 4.8%. (C) 2019 Elsevier B.V. All rights reserved.

Keyword:

Word embeddings

Contrastive learning

Negative sampling

Entropy maximization

Semantic similarity

Boosted negative sampling by quadratically constrained entropy maximization

摘要

AI总结

期刊

机构

引用论文