The effect of gradually replacing the branched alkyl side chains of a diketopyrrolopyrrole (DPP) conjugated polymer by linear side chains containing branched siloxane end groups on the photovoltaic performance of blends of these polymers with a common fullerene acceptor is investigated. With an increasing proportion of siloxane side chains, the molecular weight and solubility of the polymers decreases. While the siloxane containing polymers exhibit a higher hole mobility in field-effect transistors, their performance in solar cells is less than the polymer with only alkyl sides chains. Using grazing-incidence wide-angle X-ray scattering, transmission electron microscopy, and fluorescence spectroscopy we identify two main reasons for the reduced performance of siloxane containing polymers in solar cells. The first one is a somewhat coarser phase-separated morphology with slightly wider polymer fibers. This is unexpected as often the fiber width is inversely correlated with polymer solubility. The second one is stronger non-radiative decay of the pristine polymers containing siloxane side chains.