In this paper the usual concept of optimality in a two person zero sum Markov game is studied. A necessary but not sufficient condition for strategies to be optimal is derived, and also a sufficient but not necessary condition. The gap between these two conditions is not very wide, and can be closed quite elegantly in modifying the definition of optimality. One of these modified concepts for optimality, the so called persistent optimality, seems to be more akin to the concept of optimality in Markov decision processes. Subgame perfectness, another optimality concept, is also characterized.