An experimental survey of no-reference video quality assessment methods

Purpose- The Video Quality Metric (VQM) is one of the most used objective methods to assess video quality, because of its high correlation with the human visual system (HVS). VQM is, however, not viable in real-time deployments such as mobile deployments, not only due to its high computational demands but, because, as a Full-Reference (FR) metric, it requires both the original video and its impaired counterpart. In contrast, No-Reference (NR) objective algorithms operate directly on the impaired video and are considerably faster, but loose out in accuracy. This research studies how differently NR metrics perform in the presence of network impairments.
Design/methodology/approach- We assess 8 NR metrics, alongside a lightweight FR metric, using VQM as benchmark in a self developed network-impaired video dataset. Our study covers a range of methods, a diverse set of video types and encoding conditions, and a variety of network impairment test-cases.
Findings- We show the extent by which packet loss affects different video types, correlating the accuracy of NR metrics to the FR benchmark. Our study helps identifying the conditions under which simple metrics may be used effectively and indicates an avenue to control the quality of streaming systems.
Originality/value- Most studies in literature have focused on assessing streams that are either unaffected by the network (e.g., looking at the effects of video compression algorithms) or are affected by synthetic network impairments (i.e. via simulated network conditions). We show that when streams are affected by real network conditions, assessing QoE becomes even harder, as the existing metrics perform poorly.