.png){kind=icon}
返回
Learning to crawl deep web
.png){kind=icon}
PRE
AI
DOI：10.1016/j.is.2013.02.001.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Deep web or hidden web refers to the hidden part of the Web (usually residing in structured databases) that remains unavailable for standard Web crawlers. Obtaining content of the deep web is challenging and has been acknowledged as a significant gap in the coverage of search engines. The paper proposes a novel deep web crawling framework based on reinforcement learning, in which the crawler is regarded as an agent and deep web database as the environment. The agent perceives its current state and selects an action (query) to submit to the environment (the deep web database) according to Q-value. While the existing methods rely on an assumption that all deep web databases possess full-text search interfaces and solely utilize the statistics (TF or DF) of acquired data records to generate the next query, the reinforcement learning framework not only enables crawlers to learn a promising crawling strategy from its own experience, but also allows for utilizing diverse features of query keywords. Experimental results show that the method outperforms the state of art methods in terms of crawling capability and relaxes the assumption of full-text search implied by existing methods. (C) 2013 Elsevier Ltd. All rights reserved.
Keyword:
Hidden web
Deep web crawling
Reinforcement learning
期刊
IF:
3.9
论文数:
2.8K
被引数:
1.8K
机构
引用论文
Distributed Bayesian Computation and Self-Organized Learning in Sheets of Spiking Neurons with Local Lateral Inhibition.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAI分享收藏分享收藏
查看更多内容