.png){kind=icon}
返回
SpaceTransformers: Language Modeling for Space Systems
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2021.3115659.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The transformers architecture and transfer learning have radically modified the Natural Language Processing (NLP) landscape, enabling new applications in fields where open source labelled datasets are scarce. Space systems engineering is a field with limited access to large labelled corpora and a need for enhanced knowledge reuse of accumulated design data. Transformers models such as the Bidirectional Encoder Representations from Transformers (BERT) and the Robustly Optimised BERT Pretraining Approach (RoBERTa) are however trained on general corpora. To answer the need for domain-specific contextualised word embedding in the space field, we propose SpaceTransformers, a novel family of three models, SpaceBERT, SpaceRoBERTa and SpaceSciBERT, respectively further pre-trained from BERT, RoBERTa and SciBERT on our domain-specific corpus. We collect and label a new dataset of space systems concepts based on space standards. We fine-tune and compare our domain-specific models to their general counterparts on a domain-specific Concept Recognition (CR) task. Our study rightly demonstrates that the models further pre-trained on a space corpus outperform their respective baseline models in the Concept Recognition task, with SpaceRoBERTa achieving significant higher ranking overall.
Keyword:
Modeling
Task analysis
Bit error rate
Training
Data models
Transfer learning
Transformers
Language model
transformers
space systems
concept recognition
requirements
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
Design and performances evaluation of new Costas‐based radar waveforms with pulse coding diversity基于Costas的脉冲编码多样性新雷达波形的设计与性能评估
OAAI
OAAI分享收藏Matrix Effects in the Detection of Pb and Ba in Soils Using Laser-Induced Breakdown Spectroscopy使用激光诱导击穿光谱检测土壤中Pb和Ba的基质效应
PREAI
分享收藏Transfer Learning for Arabic Named Entity Recognition With Deep Neural Networks基于深度神经网络的阿拉伯命名实体识别的迁移学习.png)
IEEE ACCESSOAAI
IEEE ACCESSIF3.6
OAAI分享收藏分享收藏
查看更多内容