Extracting parallel fragments from comparable documents using a generative model

doi:10.1016/j.csl.2018.07.002

Extracting parallel fragments from comparable documents using a generative model

2019-01-01

1

PRE

AI

DOI：10.1016/j.csl.2018.07.002

原文链接

原文求助

摘要

En 中文

Although parallel corpora are essential language resources for many natural language processing tasks, they are rare or even not available for many language pairs. Instead, comparable corpora are widely available and contain parallel fragments of information that can be used in applications like statistical machine translation systems. In this research, we propose a generative latent Dirichlet allocation based model for extracting parallel fragments from comparable documents without using any initial parallel data or bilingual lexicon. The experimental results show significant improvement if the extracted fragments generated by the proposed method are used for augmenting an existing parallel corpus in an statistical machine translation system. According to the human judgment, the accuracy of the proposed method for an English-Persian task is about 59.7%. Also, the out of vocabulary error rate for the same task is reduced by 28%. (C) 2018 Elsevier Ltd. All rights reserved.

Keyword:

Fragment extraction

Comparable corpora

Generative model

Statistical machine translation

Persian

English

German

Extracting parallel fragments from comparable documents using a generative model

摘要

AI总结

期刊

机构