Our goal is to unravel the mechanisms that lead to failure of a ductile two-phase material - that consists of a ductile soft phase and a relatively brittle hard phase. An idealized microstructural model is used to study damage propagation systematically and transparently. The analysis uncovers distinct microstructural features around early voids, whereby regions of the hard phase are aligned with the tensile axis and regions of the soft phase are aligned with the shear directions. These features are consistently found in regions exhibiting damage propagation, whereby the damage remains initiation driven, i.e. voids nucleate independently of each other. Upon localization, damage is controlled on a longer length-scale relying on a critical relative position of 'initiation hot-spots'. The damage rapidly increases in bands of the soft phase wherein several voids are aligned with the shear directions. The relative arrangement of the voids determines whether the microstructure fails early, or at a substantially higher strain. Although much research is needed to refine these findings for real or more realistic microstructures, in particular in three-dimensions, this paper opens a route to a deeper understanding of the ultimate failure of multi-phase materials.