Abstract In this paper, we analyze the interaction between friction-induced vibrations and self-sustained lateral vibrations caused by a mass-unbalance in an experimental rotor dynamic setup. This study is performed on the level of both numerical and experimental bifurcation analyses. Numerical analyses show that two types of torsional vibrations can appear: friction-induced torsional vibrations and torsional vibrations due to the coupling between torsional and lateral dynamics in the system. Moreover, both the numerical and experimental results show that a higher level of mass-unbalance, which generally increases the lateral vibrations, can have a stabilizing effect on the torsional dynamics, i.e. friction-induced limit cycling can disappear. Both types of analysis provide insight in the fundamental mechanisms causing self-sustained oscillations in rotor systems with flexibility, mass-unbalance and discontinuous friction which support the design of such flexible rotor systems.