We present an extended finite element method (XFEM) for the direct numerical simulation of the flow of viscoelastic fluids with suspended particles. Formoving particle problems, we devise a temporary arbitrary Lagrangian-Eulerian (ALE) scheme which defines the mapping of field variables at previous time levels onto the computational mesh at the current time level. In this method, a regular mesh is used for the whole computational domain including both fluid and particles. A temporary ALE mesh is constructed separately and the computational mesh is kept unchanged throughout the whole computations. Particles are moving on a fixed Eulerian mesh without any need of re-meshing. For mesh refinements around the interface, we combine XFEM with the grid deformation method, in which nodal points are redistributed close to the interface while preserving the mesh topology. Our method is verified by comparing with the results of boundary fitted mesh problems combined with the conventional ALE scheme. The proposed method shows similar accuracy compared with boundary fitted mesh problems and superior accuracy compared with the fictitious domain method. If the grid deformation method is combined with XFEM, the required computational time is reduced significantly compared to uniform mesh refinements, while providing mesh convergent solutions. We apply the proposed method to the particle migration in rotating Couette flow of a Giesekus fluid. We investigate the effect of initial particle positions, the Weissenberg number, the mobility parameter of the Giesekus model and the particle size on the particle migration. We also show two-particle interactions in confined shear flow of a viscoelastic fluid. We find three different regimes of particle motions according to initial separations of particles.