Results of point-particle direct numerical simulations of downward gas-solid flow in smooth and rough vertical channels are presented. Two-way coupling and inter-particle collisions are included. The rough walls are shaped as fixed layers of tiny spherical particles with diameter much smaller than the viscous wall unit. The turbulence attenuation induced by the free solid particles in the gas flow is shown to be enhanced with increasing wall roughness. The so-called feedback force, the force exerted by the free particles on the gas, is decomposed into three contributions: the domain average of the mean feedback force, the non-uniform part of the mean feedback force and the fluctuating part of the feedback force. Since the non-uniformity of the mean feedback force increases with wall roughness, the effect of the non-uniform part of the mean feedback force is investigated in detail. For both smooth and rough walls, the non-uniform part of the mean feedback force is shown to contribute significantly to the particle-induced turbulence attenuation.