Speech communication in natural environments is often impaired by varying levels of ambient noise. Such noise can reduce speech intelligibility and make conversations more effortful, consequently causing an increase in arousal, frustration or stress in the partaking speakers. This contribution investigates the effects of background noise on arousal in a speech communication setting using collaborative tasks and examines the measurability of such detrimental effects through physiological signals – heart rate variability and skin conductance. We focus on the differences in responses between the various noise conditions aiming to establish the sensitivity of the employed physiological measures. Furthermore, self-reported mental effort scores are used to assess the dependency of subjective mental effort requirements on background noise for the examined communication setting. Our results indicate that while mental effort scores show a significant positive relationship with background noise level, skin conductance and heart rate variability features, which are commonly employed for arousal state detection, may be inappropriate and lack sensitivity to distinguish communication settings differing solely by the level of background noise.