A photoluminescence (PL) study is made of (001) oriented GaAs samples containing a single delta doping layer of silicon. Depending on the growth temperature several novel narrow spectral features are observed that are related to the delta doping procedure, but cannot be attributed to transitions involving conduction electron subbands. The nature of one specific PL feature at 1.4977 eV is explored in detail. It is present in delta-doped samples grown at low temperatures (480 degrees C) and shows all the characteristics of a bound exciton recombination. Its most conspicuous properties are a strong linear polarization along (110) and a (001) axially symmetric spin splitting in high magnetic fields. The whole of the spectroscopic data leads to a model in which an exciton is bound to a Si-As-Si complex, aligned along the (110) direction.