Compressed video is degraded in quality due to the introduction of coding artifacts. A two-step subjective experiment was performed to evaluate the most visible artifacts and their relation to video quality for AVS and H.264 compressed video. In the first step, non-expert viewers were requested to score the image quality degradation as a function of compression ratio for various video sequences and to indicate which artifact was perceived during scoring. During the second step, eight trained viewers were asked to score the strength of three artifacts, i.e., blurring, blocking, and color distortion, which were reported as the most perceivable artifacts in the first step of the experiment. The quality performance between AVS and H.264 was also compared. The analysis of covariance indicated that the quality performance between AVS and H.264 was very close. A linear regression analysis showed that for the CIF videos 96% of the variance in quality degradation could be predicted by linearly combining the normalized strengths of the three most visible artifacts. © 2009 Elsevier B.V. All rights reserved.