Since polymers play an increasingly important role in both structural and tribological applications, understanding their intrinsic mechanical response is key. Therefore in the last decades much effort has been devoted into the development of constitutive models that capture the polymers' intrinsic mechanical response quantitatively. An example is the Eindhoven Glassy Polymer model. In practice most polymers are filled, e.g. with hard particles or fibers, with colorants, or with soft particles that serve as impact modifiers. To characterize the influence of type and amount of filler particles on the intrinsic mechanical response, we designed model systems of polycarbonate with different volume fractions of small, order 100 nm sized, either hard or soft particles, and tested them in lubricated uniaxial compression experiments. To reveal the local effects on interparticle level, three-dimensional representative volume elements (RVEs) were constructed. The matrix material is modeled with the EGP model and the fillers with their individual mechanical properties. It is first shown that (only) 32 particles are sufficient to capture the statistical variations in these systems. Comparing the simulated response of the RVEs with the experiments demonstrates that in the small strain regime the stress is under-predicted since the polymer matrix is modeled by using only one single relaxation time. The yield- and the large strain response is captured well for the soft-particle filled systems while, for the hard-particles at increased filler loadings, the predictions are less accurate. This is likely caused by polymer-filler interactions that result in accelerated physical aging of the polymer matrix close to the surfaces. Modifying the Sa-parameter, that captures the thermodynamic state of the polymer matrix, allows to correctly predict the macroscopic response after yield. The simulations reveal that all rate-dependencies of the different filled systems originate from that of the polymer matrix. Finally, an onset is presented to predict local and global failure based on critical events on the microlevel, that are likely to cause the over-prediction in the large-strain response of the hard-particle filled systems.