Making use of droplet epitaxy, we systematically controlled the height of self-assembled GaAs quantum dots by more than one order of magnitude. The photoluminescence spectra of single quantum dots revealed the strong dependence of the spectral linewidth on the dot height. Tall dots with a height of ~30 nm showed broad spectral peaks with an average width as large as ~5 meV, but shallow dots with a height of ~2 nm showed resolution-limited spectral lines (120 µeV). The measured height dependence of the linewidths is in good agreement with Stark coefficients calculated for the experimental shape variation. We attribute the microscopic source of fluctuating electric fields to the random motion of surface charges at the vacuum-semiconductor interface. Our results offer guidelines for creating frequency-locked photon sources, which will serve as key devices for long-distance quantum key distribution.