Objective Video Quality Assessment 

Introduction

The “digital switchover” has started in United Kingdom on 17^th of October 2007, with all TV broadcasts to be switched over to digital by 2012. Similar plans are in place in most countries around the world.

The rapid progress of the digital video broadcasting and distribution has been made possible by the advances in digital video compression algorithms (most notably MPEG-2 and MPEG-4) which can achieve compression ratios of 100-to-1 and better.

The challenge now faced by video content providers and distributors is to find a balance between compression ratios (bandwidth) and video quality.

Due to the huge amount of digital video content created and distributed is becoming less and less feasible to visually check the compressed video streams (subjective testing) so new methods have been designed which allow automatic, rapid video quality assessments.

Video quality assessment methods

One method of video quality assessment is to select a number of people an ask them to give scores for the video, further processing and averaging the results can result in a mean opinion score for the video which can be used by the QA team as an indicator of the quality o the video . This method is known as subjective video quality assessment. The reliability of the results depends on the number and quality of people selected, the viewing conditions, and the processing of the individual scores. Some standards for subjective video quality have been developed like “ITU-R BT.500-11 Methodology for the subjective assessment of the quality of television pictures” ^[1].

Unfortunately, subjective video quality assessment is time-consuming and expensive. In an R&D environment, where algorithm changes need to be continually assessed or in a broadcasting environment, where the video quality monitoring should be performed in real-time, subjective video quality assessment cannot be used.

To reduce time and cost objective video quality assessment methods have been developed to automatically predict perceived video quality. These methods aim to estimate the mean opinion score found with subjective testing. Although they do not perfectly correlate with subjective scores (the visual impact on a human observer) recent methods achieve a good enough correlation to replace subjective assessment, drastically reducing time and cost of the quality assurance process. Depending on the amount of original data available the methods also known as metrics, are divided in three categories:

• Full reference metrics; metrics that need the original videos in order to assess the quality of a processed video.
• Partial reference metrics; metrics that need only some information about the original video.   
•  No reference metrics; metrics that assess the quality based solely on the processed video. 

This article focuses on full reference metrics. 

Objective video quality assessment

Classic, objective video quality assessment methods like PSNR are based on calculating, for each pixel, the absolute difference between the reference (original) video and the processed video. These absolute differences are then transformed into a quality score by using various statistical methods. 

The most popular video quality metrics based on statistical processing of absolute pixel differences are: 

•           MSE – Mean Square Error
•           SNR – Signal-to-Noise Ratio
•           PSNR – Peak Signal-to-Noise Ratio
•           CZD - Czekanowski distance
•           Minkowski Distance 

The main drawback of all the above video quality metrics is that they are not sensitive to structural or contrast differences which affect the perceived video quality. In spite of their drawbacks because of their computational efficiency, metrics like PSNR became over the time the “de facto” standard for objective video quality assessment. Recently, several efforts were made to develop metrics that would correlate better with subjective assessment.

Metrics like SSIM^[2] and VQM^[3]  are based on the properties of the Human Visual System and instead of directly comparing pixel values the metrics subtract structural information from the video frames and compare it. Their predictions are more reliable than those of pixel-difference based methods.  

Over the time several tests^[4][5] were performed to find the best metric. The tests showed that most of the pixel-difference methods had statistically equivalent performance and only a few metrics like SSIM and VQM, had a statistically better performance than PSNR. The correlation of SSIM or VQM with subjective scores is high enough to reliably replace subjective testing.  

VQLab

Semaca's VQLab (http://www.vqlab.com) is a fast, reliable and cost-effective tool for assessing the quality of your processed video. With native support for a wide range of video formats: AVI, MPEG-2, MPEG-4, AVC/H.264, VC1, DV, raw YUV format (supported fourCC: YV12, UYVY, AYUV, YUY2, YVYU, YVU9, Y211, Y41P, IYUV, CLJR, I420), VQLab can also be used to analyze any other video format supported by the DirectShow framework installed on a target machine. VQLab uses the PSNR, SSIM and CZD metrics in order to provide an accurate, objective measure of the video quality. The metric data generated can be easily displayed, sorted, filtered, compared, visualized and exported. 

In an R&D environment, VQLab allows every developer and tester to use a fast and reliable tool for measuring the video quality. Video processing algorithm development and tuning becomes a much faster and reliable process. VQLab can achieve real-time (and faster) speed allowing more test scenarios to be executed resulting in improved product quality.

VQLab's powerful Command Line Interface allows complete automation of the video quality control stage in an R&D or content creation environment resulting in reduced time-to-market and increased product quality.

Reference: