ARTS: An adaptive regularization training schedule for activation sparsity exploration

Zeqi Zhu, Arash Pourtaherian, Luc Waeijen, Lennart Bamberg, Egor Bondarev, Orlando Moreira

Onderzoeksoutput: Hoofdstuk in Boek/Rapport/CongresprocedureConferentiebijdrageAcademicpeer review

5 Citaten (Scopus)
15 Downloads (Pure)

Samenvatting

Brain-inspired event-based processors have attracted considerable attention for edge deployment because of their ability to efficiently process Convolutional Neural Networks (CNNs) by exploiting sparsity. On such processors, one critical feature is that the speed and energy consumption of CNN inference are approximately proportional to the number of non-zero values in the activation maps. Thus, to achieve top performance, an efficient training algorithm is required to largely suppress the activations in CNNs. We propose a novel training method, called Adaptive-Regularization Training Schedule (ARTS), which dramatically decreases the non-zero activations in a model by adaptively altering the regularization coefficient through training. We evaluate our method across an extensive range of computer vision applications, including image classification, object recognition, depth estimation, and semantic segmentation. The results show that our technique can achieve 1.41 × to 6.00 × more activation suppression on top of ReLU activation across various networks and applications, and outperforms the state-of-the-art methods in terms of training time, activation suppression gains, and accuracy. A case study for a commercially-available event-based processor, Neuronflow, shows that the activation suppression achieved by ARTS effectively reduces CNN inference latency by up to 8.4 × and energy consumption by up to 14.1 ×.

Originele taal-2Engels
TitelProceedings - 2022 25th Euromicro Conference on Digital System Design, DSD 2022
RedacteurenHimar Fabelo, Samuel Ortega, Amund Skavhaug
UitgeverijInstitute of Electrical and Electronics Engineers
Pagina's415-422
Aantal pagina's8
ISBN van elektronische versie978-1-6654-7404-7
DOI's
StatusGepubliceerd - 4 jan. 2023
Evenement25th Euromicro Conference on Digital Systems Design, DSD 2022 - ExpoMeloneras Convention Center, Maspalomas, Gran Canaria, Spanje
Duur: 31 aug. 20222 sep. 2022
Congresnummer: 25
https://dsd-seaa2022.iuma.ulpgc.es

Congres

Congres25th Euromicro Conference on Digital Systems Design, DSD 2022
Verkorte titelDSD 2022
Land/RegioSpanje
StadMaspalomas, Gran Canaria
Periode31/08/222/09/22
Internet adres

Bibliografische nota

Funding Information:
This publication is supported by the ANDANTE project, which has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 876925. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and France, Belgium, Germany, Netherlands, Portugal, Spain, Switzerland.

Financiering

This publication is supported by the ANDANTE project, which has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 876925. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and France, Belgium, Germany, Netherlands, Portugal, Spain, Switzerland. This publication is supported by the ANDANTE project, which has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 876925. The JU receives support from the European Union s Horizon 2020 research and innovation programme and France, Belgium, Germany, Netherlands, Portugal, Spain, Switzerland.

Vingerafdruk

Duik in de onderzoeksthema's van 'ARTS: An adaptive regularization training schedule for activation sparsity exploration'. Samen vormen ze een unieke vingerafdruk.

Citeer dit