Computational Fluid Dynamics (CFD) is increasingly used to predict wind flow and pollutant dispersion around buildings. The two most frequently used approaches are solving the Reynolds-averaged Navier–Stokes (RANS) equations and Large-Eddy Simulation (LES). In the present study, we compare the convective and turbulent mass fluxes predicted by these two approaches for two configurations of isolated buildings with distinctive features. We use this analysis to clarify the role of these two components of mass transport on the prediction accuracy of RANS and LES in terms of mean concentration. It is shown that the proper simulation of the convective fluxes is essential to predict an accurate concentration field. In addition, appropriate parameterization of the turbulent fluxes is needed with RANS models, while only the subgrid-scale effects are modeled with LES. Therefore, when the source is located outside of recirculation regions (case 1), both RANS and LES can provide accurate results. When the influence of the building is higher (case 2), RANS models predict erroneous convective fluxes and are largely outperformed by LES in terms of prediction accuracy of mean concentration. These conclusions suggest that the choice of the appropriate turbulence model depends on the configuration of the dispersion problem under study. It is also shown that for both cases LES predicts a counter-gradient mechanism of the streamwise turbulent mass transport, which is not reproduced by the gradient-diffusion hypothesis that is generally used with RANS models.