.png){kind=icon}
返回
CombTransformers: Statement-Wise Transformers for Statement-Wise Representations
.png){kind=icon}
OA
AI
DOI：10.1109/TSE.2023.3310793.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
This study presents a novel category of Transformer architectures known as comb transformers, which effectively reduce the space complexity of the self-attention layer from a quadratic to a subquadratic level. This is achieved by processing sequence segments independently and incorporating X-word embeddings to merge cross-segment information. The reduction in attention memory requirements enables the deployment of deeper architectures, potentially leading to more competitive outcomes. Furthermore, we design an abstract syntax tree (AST)-based code representation to effectively exploit comb transformer properties. To explore the potential of our approach, we develop nine specific instances based on three popular architectural concepts: funnel, hourglass, and encoder-decoder. These architectures are subsequently trained on three code-related tasks: method name generation, code search, and code summarization. These tasks encompass a range of capabilities: short/long sequence generation and classification. In addition to the proposed comb transformers, we also evaluate several baseline architectures for comparative analysis. Our findings demonstrate that the comb transformers match the performance of the baselines and frequently perform better.
Keyword:
Programming languages
machine learning
learning representations
code search and summarization
method name Gen
期刊
IF:
5.6
论文数:
2.9K
被引数:
1.1W
机构
引用论文
Matrix Effects in the Detection of Pb and Ba in Soils Using Laser-Induced Breakdown Spectroscopy使用激光诱导击穿光谱检测土壤中Pb和Ba的基质效应
.png){kind=icon}
PREAI
分享收藏Frequency stabilization of a diode laser to Doppler-free spectrum of molecular iodine at 633 nm
PREAI
分享收藏An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation通过神经机器翻译在野外学习错误修复补丁的实证研究
OAAI
OAAI分享收藏
查看更多内容