Optical damage limits the application range of congruent LiNbO3. This problem is commonly overcome by adding optical-damage-resistant cations. Here, the influence of doping with optical-damage-resistant Mg and Zn on the ionic and piezoelectric contributions to the dielectric permittivity is investigated in a broad frequency range (1 mHz-2 THz). It is shown that the two dopants have radically different influences on the variation of ionic permittivity with doping, in spite of their similarities with respect to the crystallographic structure. Raman spectroscopy reveals that the difference in permittivity can be traced to the effect of Mg and Zn doping on the susceptibility of the phonon modes. Both observations point to differences in the defect incorporation mechanisms.