.png){kind=icon}
返回
Universal structural patterns in sparse recurrent neural networks
.png){kind=icon}
OA
AI
DOI：10.1038/s42005-023-01364-0.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Sparse neural networks can achieve performance comparable to fully connected networks but need less energy and memory, showing great promise for deploying artificial intelligence in resource-limited devices. While significant progress has been made in recent years in developing approaches to sparsify neural networks, artificial neural networks are notorious as black boxes, and it remains an open question whether well-performing neural networks have common structural features. Here, we analyze the evolution of recurrent neural networks (RNNs) trained by different sparsification strategies and for different tasks, and explore the topological regularities of these sparsified networks. We find that the optimized sparse topologies share a universal pattern of signed motifs, RNNs evolve towards structurally balanced configurations during sparsification, and structural balance can improve the performance of sparse RNNs in a variety of tasks. Such structural balance patterns also emerge in other state-of-the-art models, including neural ordinary differential equation networks and continuous-time RNNs. Taken together, our findings not only reveal universal structural features accompanying optimized network sparsification but also offer an avenue for optimal architecture searching. Deep neural networks have shown remarkable success in application areas across physical sciences and engineering science and finding such networks that can work efficiently with less connections (weight parameters) without sacrificing performance is thus of great interest. In this work the authors show that a large number of such efficient recurrent neural networks display certain connectivity patterns in their structure.
Keyword:
MOTIFS
期刊
IF:
5.8
论文数:
2.8K
被引数:
9.2K
机构
引用论文
分享收藏Utilisation of complex sugars for biomass production, and low-energy lipid extraction from Chlorella vulgaris and Nannochloropsis oculata for sustainable biodiesel production利用复杂糖类进行生物质生产，以及从Chlorella vulgaris和Nannochloropsis oculata中提取低能耗脂质以实现可持续生物柴油生产
PREAI
分享收藏Multilevel structural evaluation of signed directed social networks based on balance theory.png)
SCIENTIFIC REPORTS
OAAI
SCIENTIFIC REPORTSIF3.9
OAAI分享收藏
查看更多内容