On the Expressiveness of Rational ReLU Neural Networks With Bounded Depth
- Gennadiy Averkov,
- ,
- Maximilian Merkert
- Brandenburg University of Technology,
- ,
- Technische Universität Braunschweig
Output:
Chapter in Book/Report/Conference proceeding
Conference contribution
Academic
Peer-reviewOpen access
Information
Output type
Output:
Chapter in Book/Report/Conference proceeding
Conference contribution
Academic
Peer-reviewOriginal language
EnglishPublication milestones
- Published - 22/01/2025
Publication status
Published - 22/01/2025
Publisher
OpenReview.netISBN (Electronic)
9798331320850Host publication title
13th International Conference on Learning Representations, ICLR 2025Funding Details
We would like to thank Sergey Grosman from Siemens (Konstanz, Germany) for his feedback on the practical role of quantization. The research of GA was partially supported by the DFG Project 539867386 “Extremal bodies with respect to lattice functionals”, which is carried out within the DFG Priority Program 2458 “Combinatorial Synergies”.
Related Event
Title
13th international Conference on Learning Representations, ICLR 2025
Event type
ConferenceDegree of recognition
International eventDate
24/04/2025 - 28/04/2025Location
SingaporeSingapore
Abstract
<p>To confirm that the expressive power of ReLU neural networks grows with their depth, the function F
<sub>n</sub> = max{0, x
<sub>1</sub>,..., x
<sub>n</sub>} has been considered in the literature. A conjecture by Hertrich, Basu, Di Summa, and Skutella [NeurIPS 2021] states that any ReLU network that exactly represents F
<sub>n</sub> has at least dlog
<sub>2</sub>(n+ 1)e hidden layers. The conjecture has recently been confirmed for networks with integer weights by Haase, Hertrich, and Loho [ICLR 2023]. We follow up on this line of research and show that, within ReLU networks whose weights are decimal fractions, F
<sub>n</sub> can only be represented by networks with at least dlog
<sub>3</sub>(n + 1)e hidden layers. Moreover, if all weights are N-ary fractions, then F
<sub>n</sub> can only be represented by networks with at least Ω(
<sub>ln</sub>
<sup>ln</sup>
<sub>ln</sub>
<sup>n</sup>
<sub>N</sub> ) layers. These results are a partial confirmation of the above conjecture for rational ReLU networks, and provide the first non-constant lower bound on the depth of practically relevant ReLU networks.</p>
Documents and Links
Final published version
Metrics
Metrics
Scopus
citations
5
SciVal
FWCI
2.79
SciVal
Author count
3
SciVal
Paper percentile
91
SciVal
citations
5
SciVal
Top percentile
10
Download statistics
Download count
137
PlumX
Citation count
5
