We propose that the collision rates of non-spherical particles settling in a turbulent environment are significantly higher than those of spherical particles of the same mass and volume. The theoretical argument is based on the dependence of the particle drag force on the particle orientation, thus varying gravitational settling velocities, which can remain different until contact due to the particle inertia. Therefore, non-spherical particles can collide with large relative velocities. Direct numerical simulations (DNS) of streamwise decaying isotropic turbulence seeded with small, heavy, rotationally symmetric ellipsoids of five different aspect ratios are performed to confirm these arguments. The motion of 21 million ellipsoids is tracked by a Lagrangian particle solver assuming creeping flow conditions and neglecting the influence of the particles on the flow. We find that ellipsoids collide considerably more often than spherical particles of the same volume and mass due to a drastically increased mean relative velocity at contact.