Skip to search boxSkip to navigationSkip to main content

On the Expressiveness of Rational ReLU Neural Networks With Bounded Depth

Output:
Chapter in Book/Report/Conference proceeding
Conference contribution
Academic
Peer-review

Open access

Information

Output type

Output:
Chapter in Book/Report/Conference proceeding
Conference contribution
Academic
Peer-review

Original language

English

Publication milestones

  • Published - 22/01/2025

Publication status

Published - 22/01/2025

Publisher

OpenReview.net

ISBN (Electronic)

9798331320850

Host publication title

13th International Conference on Learning Representations, ICLR 2025

Funding Details

We would like to thank Sergey Grosman from Siemens (Konstanz, Germany) for his feedback on the practical role of quantization. The research of GA was partially supported by the DFG Project 539867386 “Extremal bodies with respect to lattice functionals”, which is carried out within the DFG Priority Program 2458 “Combinatorial Synergies”.

Related Event

Title

13th international Conference on Learning Representations, ICLR 2025

Event type

Conference

Degree of recognition

International event

Date

24/04/2025 - 28/04/2025

Location

SingaporeSingapore

Abstract

To confirm that the expressive power of ReLU neural networks grows with their depth, the function F n = max{0, x 1,..., x n} has been considered in the literature. A conjecture by Hertrich, Basu, Di Summa, and Skutella [NeurIPS 2021] states that any ReLU network that exactly represents F n has at least dlog 2(n+ 1)e hidden layers. The conjecture has recently been confirmed for networks with integer weights by Haase, Hertrich, and Loho [ICLR 2023]. We follow up on this line of research and show that, within ReLU networks whose weights are decimal fractions, F n can only be represented by networks with at least dlog 3(n + 1)e hidden layers. Moreover, if all weights are N-ary fractions, then F n can only be represented by networks with at least Ω( ln ln ln n N ) layers. These results are a partial confirmation of the above conjecture for rational ReLU networks, and provide the first non-constant lower bound on the depth of practically relevant ReLU networks.

Metrics

Metrics

Scopus
citations
5
SciVal
FWCI
2.59
SciVal
Author count
3
SciVal
Paper percentile
90
SciVal
citations
5
SciVal
Top percentile
10
Download statistics
Download count
143

PlumX

Citation count
5