Auto-FSDformer: Searching for fully spike-driven transformer

2026-08-08

PRE

Yang An

Ying Liu *

DOI：10.1016/j.knosys.2026.116805

Original

Original request for help

Save

Abstract

En 中文

Spiking Neural Networks (SNNs), which enable low-power, spike-driven computation through bio-inspired spiking neurons, have attracted significant attention recently. Transformer-based SNNs, often referred to as spiking transformers, have also been proposed, in which various spike-driven self-attention (SDSA) mechanisms have been exploited to alleviate the heavy computational burden associated with standard self-attention. However, some spiking transformers still retain some traditional ANN-inspired designs that cannot operate in a fully spike-driven manner within SNNs, thereby failing to fully exploit the energy efficiency of SNNs. Considering this, we propose a zero-shot Neural Architecture Search (NAS) method, named Auto-FSDformer, to automatically design optimal fully spike-driven transformer architectures. Specifically, we design a novel spike-driven transformer search space that incorporates searchable SDSA mechanisms. To facilitate effective and efficient search within this challenging search space, we then propose a robust, training-free NASWOT-LTD proxy that leverages activation pattern diversity to represent model performance. In this proxy, we introduce a Layer and weighted Temporal Decoupling (LTD) rule to highlight the non-equivalence of activation patterns across different layers and time steps. We conduct comprehensive experiments on both static and neuromorphic classification datasets. The searched FSDformer models achieve competitive performance compared to state-of-the-art spiking transformers.

Keywords:

Spiking neural network

Spiking transformer

Zero-shot neural architecture search

Auto-FSDformer: Searching for fully spike-driven transformer

Abstract

Journal

Organization

Cited Papers

Citing Papers