Although researchers have repeatedly shown that the meaning of the same concept can vary across different contexts, it has proven difficult to predict when people will assign which meaning to a concept, and which associations will be activated by a concept. Building on the affective theory of meaning (Osgood, Suci, & Tannenbaum, 1957) and the polarity correspondence principle (Proctor & Cho, 2006), we propose the dimension-specificity hypothesis with the aim to understand and predict the context-dependency of cross-modal associations. We present three sets of experiments in which we use the dimension-specificity hypothesis to predict the cross-modal associations that should emerge between aggression-related concepts and saturation and brightness. The dimension-specificity hypothesis predicts that cross-modal associations emerge depending upon which affective dimension of meaning (i.e., the evaluation, activity, or potency dimension) is most salient in a specific context. The salience of dimensions of meaning is assumed to depend upon the relative conceptual distances between bipolar opposed concept pairs (e.g., good vs. bad). The dimension-specificity hypothesis proposes that plus and minus polarities will be attributed to the bipolar concepts, and associations between concrete and affective abstract concepts that share plus or minus polarities will become activated. Our data support the emergence of dimension-specific polarity attributions. We discuss the potential of dimension-specific polarity attributions to understand and predict how the context influences the emergence of cross-modal associations.