Measuring Image Quality

A simple metric, the Peak-Signal-to-Noise-Ratio (PSNR) [45], is adopted to measure image quality. The basic idea is to compute the PSNR metric for each of a number of frames captured at the client side, using

(a) Before Shooting (b) Start Shooting

Figure 3.8: The Triggering Event Used to Record Response Delay in AssaultCube

the difference between it and the corresponding frame captured from the original game video at the server side. In other words, the PSNR method quantifies the amount of error (noise) in the reconstructed video. The PSNR is derived using the mean squared error (MSE) in relation to the maximum possible value of the luminance (here the value is 255) as shown in equation (3.1) and equation (3.2). Here fi,jis the original signal

at pixel (i, j), Fi,j is the reconstructed signal, and M × N is the picture size. The result is a single number

in decibels. MSE = N X j=1 M X i=1 (f_i,j− Fi,j)2 ! M × N (3.1) PSNR = 10 log 255 2 MSE (3.2)

In the experiments, the pre-rendered intro movie of LEGO Batman 2 is chosen to record with the resolution 1280×800 for calculating image quality. To obtain accurate samples for image quality, a function of capturing frames is inserted in GamingAnywhere to capture a deterministic sequence of uncompressed frames from the bare metal system and GPU virtualization instance. In the experiments, no compression is performed on the captured frames for the sake of simplicity. Therefore, the original frames are captured and save as BMP file format without any compression. The PSNR method is used in the experiments to calculate the image quality. The video captured from the bare-metal system is considered the original video, while the one captured from a GPU virtualization instance is considered the reconstructed video as the GPU virtualization it adopts may cause the loss of the image quality. The PSNR values for image quality are calculated by comparing the sequence captured from the bare-metal system and the one from a tested GPU virtualization instance. The algorithm of calculating the image quality for each GPU virtualization solution is described as follows.

1. Play the pre-rendered introduction movie of LEGO Batman 2 on the bare-metal system and the tested GPU virtualization instance (for example the vDGA instance) respectively.

2. Capture the generated frames during the play of the intro movie.

3. Automatically find the same frame from the bare-metal system and the tested GPU virtualization instance, then obtain a PSNR value from these two frame.

4. Obtain 100 PSNR values by repeating the step 3 to process a deterministic sequence of uncompressed frames that is captured from the bare-metal system and the tested GPU virtualization instance.

Chapter 4

Experimental Results and Discussion

All the experimental results are discussed in this chapter. Section 4.1 provides the baseline configuration resource usage. Section 4.2 describes the results of each single GPU virtualization instance, and Section 4.3 discusses the results of double GPU virtualization instances for the test of potential scalability. The graphics card benchmarks are run fifteen times in each tested configuration, and the average and standard deviation are calculated. As for cloud games, 50 response delay samples are collected for each tested game in each tested configuration. Moreover, to calculate image quality, 100 frame samples are captured from the pre-rendered intro movie of LEGO Batman in each tested configuration.

4.1

Baseline

As mentioned in the previous chapter, the pure hardware resource consumption of each graphics card bench- mark is calculated using its total consumption minus the baseline. It is not reasonable to compare the hardware consumption of bare-metal system with that of GPU virtualization instances as they are config- ured with different hardware. Therefore, the hardware resource consumption is analyzed in two aspects. Firstly, the hardware consumption of the physical machine (Machine 2) is recorded when it runs the same graphics card benchmark locally (bare-metal system) and remotely on GPU virtualization instances via VMware Horizon View (GPU virtualization instances). These results are compared to find the pattern of hardware consumption of the physical machine for running each tested configuration. Secondly, the hardware consumption of each GPU virtualization instance is also recorded. These results are compared and analyzed to find the main bottleneck in each tested GPU virtualization instance. The expected pattern of hardware consumption of each GPU virtualization instance is that the more hardware resource an instance consumes, the better performance it should achieve.

Table 4.1 shows the baseline hardware resource consumption of each tested configuration. This baseline data is calculated from three one-minute records of hardware consumption of each tested configuration (3×60 samples in total) when running no benchmark. This data is used to calculate pure hardware consumption of each tested configuration. To ensure pure hardware consumption of each benchmark in each tested configura- tion is objectively measured, the samples of baseline and the total hardware consumption of each benchmark in each tested configuration are obtained in the same run.

From the results, we can see that the hardware consumption in each tested configuration looks stable except for the GPU usage of the bare-metal system and the CPU usage of the vSGA instance, where the standard deviation is larger than the mean. This is because, in the bare-metal system, the GPU remains almost idle all the time as no benchmark or tested game is running during the period of measuring the baseline of the hardware consumption. Nevertheless, there are few samples which GPU is not idle, causing the standard deviation higher than the mean. Similarly, the CPU usage of the vSGA instance remains between 2% and 3% for the most of the time, but there are some samples which CPU usage is much higher the average, causing the same problem.

Table 4.1: Base Hardware Resource Consumption of each Tested Configuration Tested Configuration Bare-

metal vSGA vGPU K120Q vGPU K140Q vGPU K180Q vDGA GPU (%) Mean 0.1 1.3 9.7 9.7 9.6 11.2 Std. Dev. 0.34 0.88 0.48 0.47 0.48 0.90 Graphics Memory (MB) Mean 99.1 199.9 156.3 188.3 380.3 143.3 Std. Dev. 0.03 0.35 0.00 0.00 0.00 0.00 CPU (%) Mean 26.8 2.3 3.5 2.7 3.1 3.8 Std. Dev. 8.53 7.51 3.43 3.39 3.34 3.46 RAM (MB) Mean 1,405.3 1035.1 852.0 919.6 866.9 1,002.5 Std. Dev. 17.47 10.09 11.80 8.34 14.62 8.68