Statistical mechanics-based coarse graining is used for constitutive modeling of finite elasto-viscoplastic deformation behavior of transversely isotropic materials, without relying on the associated flow rule or classical yield criteria. It was shown previously (Hütter and Tervoort, 2008c) that a detailed expression for the plastic velocity gradient can be obtained in terms of correlations of the fluctuations of the elastic deformation gradient. In this paper, we demonstrate that it is crucial to include cross-correlations between fluctuations of different components of the elastic deformation gradient in order to describe materials with strong plastic anisotropy. Subsequently, the expression for the equivalent stress is obtained from a steady-state analysis during plastic deformation, and is shown to coincide with the Hill equivalent stress in the limit of small elastic deformations. In this case, only two material parameters describe in a self-consistent manner the anisotropic structure of both the plastic velocity gradient and the equivalent stress. Finally, the new constitutive viscoplastic description is successfully applied to describe experimental yield data of oriented isotactic polypropylene, specifically the influence of anisotropy and deformation rate on yielding. Particularly, a step-by-step procedure is given to identify all model parameters.