A probabilistic model derived term weighting scheme for text classification

doi:10.1016/j.patrec.2018.03.003

A probabilistic model derived term weighting scheme for text classification

2018-07-01

19

PRE

AI

DOI：10.1016/j.patrec.2018.03.003

Original

Original request for help

Save

Abstract

En 中文

Term weighting is known as a text presentation strategy to assign appropriate value to each term to improve the performance of text classification in the task of transforming the content of textual document into a vector in the term space. Supervised weighting methods using the information on the membership of training documents in predefined classes are naturally expected to provide better results than the unsupervised ones. In this paper, a new weighting scheme is proposed via a matching score function based on a probabilistic model. We introduce a latent variable to indicate whether a term contains text classification information or not, specify conjugate priors and exploit the conjugacy by integrating out the latent indicator and the parameters. Then the non-discriminating terms can be assigned weights close to 0. Experimental results using kNN and SVM classifiers illustrate the effectiveness of the proposed approach on both small and large text data sets. (C) 2018 Published by Elsevier B.V.

Keywords:

Latent feature selection indicator

Matching score function

Naive Bayes

Supervised term weighting

Text classification

A probabilistic model derived term weighting scheme for text classification

Abstract

AI Summary

Journal

Organization