The Fourier modal method (FMM) is a method for efficiently solving Maxwell’s equations with periodic boundary conditions. In order to apply the FMM to non-periodic structures, perfectly matched layers need to be placed at the periodic boundaries and the Maxwell equations have to be formulated in terms of a contrast (scattered) field. This reformulation modifies the structure of the resulting linear systems and makes the direct application of available stable recursion algorithms impossible. We adapt the well-known S-matrix algorithm for use with the aperiodic Fourier modal method in contrast field formulation (aFMM-CFF). To this end, stable recursive relations are derived for linear systems with non-homogeneous structure. The stability of the algorithm is confirmed by numerical results.