ARTS: An adaptive regularization training schedule for activation sparsity exploration

Zeqi Zhu, Arash Pourtaherian, Luc Waeijen, Lennart Bamberg, Egor Bondarev, Orlando Moreira

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

5 Citations (Scopus)
4 Downloads (Pure)

Abstract

Brain-inspired event-based processors have attracted considerable attention for edge deployment because of their ability to efficiently process Convolutional Neural Networks (CNNs) by exploiting sparsity. On such processors, one critical feature is that the speed and energy consumption of CNN inference are approximately proportional to the number of non-zero values in the activation maps. Thus, to achieve top performance, an efficient training algorithm is required to largely suppress the activations in CNNs. We propose a novel training method, called Adaptive-Regularization Training Schedule (ARTS), which dramatically decreases the non-zero activations in a model by adaptively altering the regularization coefficient through training. We evaluate our method across an extensive range of computer vision applications, including image classification, object recognition, depth estimation, and semantic segmentation. The results show that our technique can achieve 1.41 × to 6.00 × more activation suppression on top of ReLU activation across various networks and applications, and outperforms the state-of-the-art methods in terms of training time, activation suppression gains, and accuracy. A case study for a commercially-available event-based processor, Neuronflow, shows that the activation suppression achieved by ARTS effectively reduces CNN inference latency by up to 8.4 × and energy consumption by up to 14.1 ×.

Original languageEnglish
Title of host publicationProceedings - 2022 25th Euromicro Conference on Digital System Design, DSD 2022
EditorsHimar Fabelo, Samuel Ortega, Amund Skavhaug
PublisherInstitute of Electrical and Electronics Engineers
Pages415-422
Number of pages8
ISBN (Electronic)978-1-6654-7404-7
DOIs
Publication statusPublished - 4 Jan 2023
Event25th Euromicro Conference on Digital Systems Design, DSD 2022 - ExpoMeloneras Convention Center, Maspalomas, Gran Canaria, Spain
Duration: 31 Aug 20222 Sept 2022
Conference number: 25
https://dsd-seaa2022.iuma.ulpgc.es

Conference

Conference25th Euromicro Conference on Digital Systems Design, DSD 2022
Abbreviated titleDSD 2022
Country/TerritorySpain
CityMaspalomas, Gran Canaria
Period31/08/222/09/22
Internet address

Bibliographical note

Funding Information:
This publication is supported by the ANDANTE project, which has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 876925. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and France, Belgium, Germany, Netherlands, Portugal, Spain, Switzerland.

Funding

This publication is supported by the ANDANTE project, which has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 876925. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and France, Belgium, Germany, Netherlands, Portugal, Spain, Switzerland. This publication is supported by the ANDANTE project, which has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 876925. The JU receives support from the European Union s Horizon 2020 research and innovation programme and France, Belgium, Germany, Netherlands, Portugal, Spain, Switzerland.

Keywords

  • activation sparsification
  • computation efficiency
  • deep learning
  • efficient training
  • energy reduction

Fingerprint

Dive into the research topics of 'ARTS: An adaptive regularization training schedule for activation sparsity exploration'. Together they form a unique fingerprint.

Cite this