.png){kind=icon}
.png){kind=icon}
Return
Broadcast news story segmentation using sticky hierarchical dirichlet process
.png){kind=icon}
PRE
AI
DOI：10.1007/s10489-021-03098-4.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Hidden Markov model (HMM) is a popular technique for story segmentation, where hidden Markov states represent the topics. The number of hidden states has to set manually, however, this number is often unknown. This paper proposed a nonparametric approach, called SHDP-HMM, to address this problem. By defining an HDP prior distribution on transition matrices over countably infinite state spaces, SHDP-HMM can infer the number of hidden states from the data automatically. Besides, to better model the duration of topics, we utilize a parameter for self-transition bias that reduces the transition probabilities among redundant hidden states. Given a stream of text, a Gibbs sampler labels the hidden states with topic classes. The position where the topic shifts indicates a story boundary. Experiments show that the proposed SHDP-HMM approach outperforms the traditional HMM-based approaches, and the number of hidden states can be automatically inferred from data.
Keywords:
Story segmentation
Non-parametric
HDP prior
SHDP-HMM
Infinite state spaces
Journal
IF:
3.5
Papers:
7.5K
Citations:
1.7W