In this paper we address the problem of how to make conversational agents socially aware. State-of-the-art conversational agents cannot deal with multi-user situations where the user-system dialog is interleaved with discussions between users. We describe the development of an algorithm for determining addressee-hood of user utterances. The algorithm makes errors in determining addressee-hood for individual utterances, classifying utterances intended for the system as utterances intended for the other user and the other way around, creating unexpected situations in the agent's behaviour. This raises the question of how to design the conversational agent so that the users understand why the agent behaves in particular ways. We describe a study aimed at obtaining guidelines for the design of a socially aware conversational agent. We conclude that principles for modelling the behaviour of the agent are to be derived from theories about human communication rather than from theories about human-computer interaction.