In developing organic semiconductor polymers for photovoltaic applications, chlorine substitution has become an effective strategy in replacing fluorine substitution to overcome the drawbacks of low yield and high cost, commonly associated with fluorination. In general, several molecular positions are available for chlorination. To obtain a clear understanding of the impact of chlorine substitution on the intrinsic polymer properties, an investigation of structure-property relationships is necessary. Herein, four donor-acceptor type polymers with the same conjugated backbone and flexible alkyl chains, but with chlorine atoms in different positions, are employed to systematically investigate the effect of the site of chlorination on the optoelectronic properties and photovoltaic performance. Substitution of fluorine by chlorine in the backbone slightly increases open circuit voltage (Voc) and fill factor (FF) of the solar cells but causes a loss of short-circuit current density (Jsc). The introduction of chlorine in the conjugated side chains, however, significantly improves Voc, FF, and power conversion efficiency, benefiting from a lower HOMO energy level, efficient and well-balanced transport properties, and superior nanoscale morphology.