TinyFormer paper is accepted by IEEE TCAS-I 2025!

👏 Paper title: TinyFormer: Efficient Sparse Transformer Design and Deployment on Tiny Devices. In this paper, we propose TinyFormer, a framework specifically designed to develop and deploy resource-efficient transformer models on Microcontrollers (MCUs). TinyFormer integrates architecture search (SuperNAS), sparse model optimization (SparseNAS), and an automated deployment tool (SparseEngine). Experimental results on CIFAR-10 demonstrate that TinyFormer can design efficient transformers achieving 96.1% accuracy while adhering to strict hardware constraints (1MB storage and 320KB memory), delivering up to 12.2× inference speedup compared to the CMSIS-NN library. [related project]