Dual phase (DP) steels provide good strength and ductility properties. Nevertheless, their forming capability is limited due to the damage characteristics of their constituting microstructural phases and interfaces. In this work, a specific type of interface is analysed, i.e., prior austenite grain boundaries (PAGBs). In the literature, prior austenite grain boundary fracture has been reported as an important damage mechanism of DP-steels. The influence of the morphology of phase boundaries near the PAGB and the role of the martensite substructure in the vicinity of a PAGB on damage initiation is analysed. The experimentally observed preferred sites of crack nucleation along the PAGB are assessed and clarified. A finite strain rate dependent crystal plasticity model accounting for the anisotropic elasto-plasticity of martensite (and also ferrite) was applied to an idealized volume element approximating a typical small-scale PAGB microstructure. The boundary value problem is solved using a fast Fourier transform (FFT) based spectral solver. The role of crystallography and geometrical features within the volume element is studied using simulations. Results are discussed considering possibly dominant regimes of elasticity and plasticity.