The influence has been studied of the ionization laser polarization on the effective temperature of an ultracold electron source, which is based on near-threshold photoionization. This source is capable of producing both highintensity and high-coherence electron pulses, with applications in, for example, electron diffraction experiments. For both nanosecond and femtosecond photoionization, a sinusoidal dependence of the temperature on the polarization angle has been found. For most experimental conditions, the temperature is minimal when the polarization coincides with the direction of acceleration. However, surprisingly, for nanosecond ionization, a regime exists when the temperature is minimal when the polarization is perpendicular to the acceleration direction. This shows that in order to create electron bunches with the highest transverse coherence length, it is important to control the polarization of the ionization laser. The general trends and magnitudes of the temperature measurements are described by a model, based on the analysis of classical electron trajectories; this model further deepens our understanding of the internal mechanisms during the photoionization process. Furthermore, for nanosecond ionization, charge oscillations as a function of laser polarization have been observed; for most situations, the oscillation amplitude is small.