A model for a mode group diversity multiplexing (MGDM) transmission system is derived. The factors that may limit MGDM transmission are identified. An optical method to reduce the spatial overlap among the fields of the detected mode groups could mitigate impairments that may occur in MGDM transmission. Mode-selective spatial filtering (MSSF) is such an optical method. Experimental results support that a robust five-channel MGDM system could be realized with MSSF, as long as modal noise is limited. MSSF facilitates the combination of MGDM with wavelength-division multiplexing.