Metal halide perovskites are promising materials for future optoelectronic applications. One intriguing property, important for many applications, is the tunability of the band gap via compositional engineering. While experimental reports on changes in absorption or photoluminescence show rather good agreement for different compounds, the physical origins of these changes, namely the variations in valence and conduction band positions, are not well characterized. Here, we determine ionization energy and electron affinity values of all primary tin- and lead-based perovskites using photoelectron spectroscopy data, supported by first-principles calculations and a tight-binding analysis. We demonstrate energy level variations are primarily determined by the relative positions of the atomic energy levels of metal cations and halide anions and secondarily influenced by the cation-anion interaction strength. These results mark a significant step towards understanding the electronic structure of this material class and provides the basis for rational design rules regarding the energetics in perovskite optoelectronics.