Multi-phase materials are key for modern engineering applications. They are generally characterized by a high strength and ductility. Many of these materials fail by ductile fracture of the, generally softer, matrix phase. In this work we systematically study the influence of the arrangement of the phases by correlating the microstructure of a two-phase material to the onset of ductile failure. A single topological feature is identified in which critical levels of damage are consistently indicated. It consists of a small region of the matrix phase with particles of the hard phase on both sides in a direction that depends on the applied deformation. Due to this configuration, a large tensile hydrostatic stress and plastic strain is observed inside the matrix, indicating high damage. This topological feature has, to some extent, been recognized before for certain multi-phase materials. This study however provides insight in the mechanics involved, including the influence of the loading conditions and the arrangement of the phases in the material surrounding the feature. Furthermore, a parameter study is performed to explore the influence of volume fraction and hardness of the inclusion phase. For the same macroscopic hardening response, the ductility is predicted to increase if the volume fraction of the hard phase increases while at the same time its hardness decreases.