Over the course of working on the components that make up this dissertation, we have learnt valuable lessons that apply to performance characterization on the edge of the Internet, particularly using large-scale infrastructure. We list the important ones below.
Lesson 1 Last-mile characterization is hard. The sheer variety of technologies that exist in the last mile, and the applications that people use, makes it non-trivial to adequately characterize performance.
It is worth reiterating a point that has been made multiple times in this dissertation: performance characterization in the last mile is more than any one parameter. Working on this problem forces us to step outside the comfort zone of any one network layer that we might be proficient in. For example, wireless network subtleties force us to understand how the MAC and physical layers work, while understanding application performance require us to delve deep into the application in question. Ultimately, the question of how these things interact and affect user experience requires us to understand end-to-end principles. And then, of course, there is the challenge of actually presenting this information to the users in a form that they will understand. One approach that we have proposed consist of providing a “nutrition label” to the user that would help break down the properties of the network so that the user can understand it better and make informed decision about their choice of ISP and service plan. The label consists of a suite of parameter values that cover different use cases [156]. Similar approaches might be required to ensure that users understand what is happening inside their network.
Lesson 2 A good vantage point is vital for good characterization. In particular, an available vantage point that provides an unobstructed view of the last-mile enables us to approach characterization in a more structured way.
One reason why characterizing the last mile is difficult is because data from good vantage points has been traditionally hard to obtain. The complexity and heterogeneity of the last mile makes it essential to have a vantage point that can minimize the number of
confounding factors that affect characterization. The gateway serves as a good vantage point in this situation, but it is also important to note that it is not a magic pill; it is a light-weight, resource-constrained device with limited memory and computational power. These constraints limit the kind of measurements we can collect; e.g., involved application performance diagnosis may not be possible due to limits to the amount of data we can collect and process on the device. The lack of visibility into the Internet beyond the ISP is also a drawback. Ultimately, understanding the limitations of current approaches and developing new methods for tackling issues is critical in the rapidly changing environment that is the modern Internet.
Lesson 3 New settings call for new measurement metrics and techniques. It is sometimes beneficial to extend old techniques and metrics, and maybe even define new ones to suit the new setting.
The uniqueness of the gateway as a vantage point, as well as its shortcomings, made it necessary to rethink how we characterize the last mile. In Chapter 5, we go beyond regular throughput and latency measurements and show how newer techniques such as last mile latency, and latency under load provide more insight. In Chapter 6, we describe how to detect bottlenecks using passively collected data under severe hardware constraints, while in Chapter 7 we use a Web browser emulator that is targeted towards identifying network bottlenecks. We also implement optimizations to improve performance on the gateway. The above techniques would be either not possible or not required in a different setting, but in the gateway they allow us to gain new insight into last-mile performance.
Lesson 4 Reappraising conventional wisdom gives us new insight. With the rate at which the Internet keeps changing, a fresh look at accepted truths can throw up surprises.
When we started out with trying to understand Web performance, we were faced with a body of work that went back more than a decade. However, we had a unique viewpoint that previous work did not have. Our work in Chapter 7 reaffirms a lot of the previous work — it showed how latency is the primary bottleneck especially as throughput goes up — but
our data offered new insight into the nature of the latency and throughput bottlenecks in the last mile. We were able to tease out the contribution of various latency components in a way that had not been done before. We were also able to test optimizations that had been proposed before but in a different setting, i.e., in the home and characterize how they can provide even more benefit.
Lesson 5 Deployment in real homes is hard. Technical challenges exist, but real-world constraints and the social aspect make us rethink the concept of “scale” when it comes to such deployments.
The goal of having a deployment in the form of devices that people use in their day to day lives proved more challenging than the mere development of a stable platform. Finding trustworthy volunteers, who in turn trusted us to not affect their home network or violate their privacy, and engaging with them so that they have an incentive to help us keep the deployment up, proved challenging. Designing experiments that could collect good data without affecting performance for the user or violating their privacy, while also being mindful of the resource constraints was another major challenge. We had to go through multiple revisions of hardware, firmware, software, data collection and privacy policies during the course of developing and deploying the platform.