The interrelation between the elasto-viscoplastic behavior of anisotropic solids on the macroscopic scale and the microscopic dynamics of their constituent atoms and molecules is examined. To that end, we employ a scheme for coarse graining as used in the context of the general equation for the nonequilibrium reversible-irreversible coupling (GENERIC) framework. First, the framework is introduced and illustrated with several examples that are self-contained on a single level of description, that is, that do not establish any links to other levels of description. Second, as a prototype example of applying the coarse graining scheme, the derivation of the evolution equations for nonisothermal hydrodynamics based on the microscopic Hamiltonian point mechanics is illustrated, leading to the well-known Navier-Stokes equation and the balance equations for mass and energy. Third, in the main part, we elaborate in detail on the application of the same methodology of coarse graining to elasto-viscoplastic solids. On the macroscopic scale, the elastic part of the deformation gradient is used as an internal variable to describe the state of deformation. Viscoplasticity then follows from relaxation of the elastic deformation gradient and is conveniently expressed in terms of so-called plastic velocity gradient tensor. Typically, constitutive relations for the plastic velocity gradient tensor rely on phenomenological macroscopic arguments, resulting in a large number of material constants. In this work, we illustrate a procedure to relate the plastic velocity gradient tensor to the rapid microscopic fluctuations of the elastic deformation gradient. In this way, we are able to use microscopic information about anisotropic solids to restrict the tensorial structure of the plastic velocity gradient tensor, thereby drastically reducing the number of material parameters. The antisymmetric part of the plastic velocity gradient tensor, the so-called plastic spin, naturally arises in our treatment and does not require any special constitutive assumptions.