Application Architecture Analysis

As discussed in Chapter 4, application architecture has a large impact on network heartbeats. P2P heartbeats are mostly conducted over UDP in the user/dynamic port ranges. Non-P2P heartbeats have a more diverse profile. However, we have only considered these classes in a holistic context.

Thus, we have not developed an appreciation for how individual applications make use of network heartbeats. In this section, we conduct an in-depth analysis of the different application architectures by comparing and contrasting the heartbeats produced by particular applications and vendors.

5.4.1 P2P Analysis

For analysis of P2P applications, we have examined the four P2P applications that we identified.

Two of these applications (BitTorrent and PPStream) are benign, and two (ZeroAccess and Sality) are malicious in nature. BitTorrent is a file sharing application that allows users to share files in a P2P network³. PPStream is a P2P video streaming application that originated in China and caters to an East Asian audience⁴. ZeroAccess is a well-known P2P botnet that is used primarily for

3http://www.bittorrent.com/

4http://www.pps.tv/

bitcoin mining and click fraud⁵. Sality is another P2P botnet that has a variety of functions such as conducting DDoS attacks or distributing other malware [13].

Figure 5.8 shows the protocol/port profile for each application. In addition, the hashed areas represent the number of dead heartbeats in each category.

Figure 5.8: Analysis of P2P heartbeats.

BitTorrent is the more prevalent of the benign P2P applications. As demonstrated in Figure 5.8, BitTorrent produced more heartbeats overall than PPStream or ZeroAccess. This is sensible since BitTorrent is a widely used and generic protocol that appeals to a number of users [35]. These heartbeats are split between the user and dynamic port ranges, though the majority of traffic is conducted on the user ports. This is a consequence of different implementations of BitTorrent using different port ranges. In both of these ranges, over half of all the heartbeats produced are dead. This indicates that there is a high amount of churn in BitTorrent networks, resulting in many dead heartbeats.

Relative to BitTorrent, PPStream heartbeats are less populous. From this, one can infer that

5https://www.symantec.com/security_response/writeup.jsp?docid=2011-071314-0410-99

PPStream is less widely used than BitTorrent. This inference is further supported by the niche nature of the PPStream application. Like in BitTorrent, the majority of the heartbeats are dead, however dead heartbeats make up a larger portion in this case. This indicates that PPStream too is exposed to a large amount of churn.

Figure 5.8 shows that almost all heartbeats related to the ZeroAccess botnet are dead. Ze-roAccess is a well-known botnet that has an easily identifiable signature in its network commands.

In addition, all dead ZeroAccess heartbeats are inbound, indicating that external hosts are trying to reach internal hosts. Thus, it is likely that the communications are dropped by the network’s security infrastructure.

A natural question that arises is why some internal hosts are on a neighbour’s list if they cannot communicate back. There are two possible reasons for this. One is that it may be an artifact of infection. This means that a machine was (or is) infected, but the affected host has either been cleaned, or updated firewall rules have caused the communications to be dropped. The other is that the botnet is trying to discover new neighbours by randomly making requests. The former is more likely, since the majority of this traffic is focused on a small set of hosts. We also observed one of these heartbeats from a host on our network directed to another host. The receiving host was reported as infected with ZeroAccess in the Virus Tracker database⁶. In addition, if the botnet was attempting to discover new neighbours, then it would likely scan a broader portion of the address space than what was observed.

Sality is the most unique P2P application that we have identified. As shown in Figure 5.8, Sality makes use of TCP and UDP, while the other P2P applications make use of only UDP. Combined, these heartbeats make Sality the top heartbeat generating application we observed. Since Sality has a P2P architecture, each infected host generates multiple heartbeats. And so, even though over 50,000 heartbeats are affiliated with Sality, only 239 unique hosts exhibited this behaviour. Packet captures reveal that the contents of the communications are the same, but the botnet uses both pro-tocols. The use of dual protocols could be to maximize the chances of successful communications.

6

Another notable difference is that the heartbeats of Sality are almost all alive. This is due to the way the botnet is set up. In our dataset, the infected hosts on our network only communicate with globally distributed hosts located in data centers. Thus, it does not rely on peers hosted on personal machines. This means that churn is less likely, as is the case with other P2P networks. This also supports the notion that the high proportion of other (i.e., non-Sality) dead P2P heartbeats is due to churn.

5.4.2 Non-P2P Analysis

The majority of non-P2P heartbeats were related to software or service vendors. Conducting an in-depth analysis on the hundreds of organizations/applications that produce or receive heartbeats is beyond the scope of this thesis. Thus, we selected five representative software/service vendors for in-depth analysis. These vendors represent the major contributors of non-P2P heartbeats.

We observed four of these major contributors: software vendors, service vendors, managed hosting vendors, and CDNs. Note that though there are five vendors, each one belongs to one or more of these categories.

Software vendors are organizations that sell software. Since software vendors are the producers of software they typically also design the applications that they sell. This means that any heartbeat in the application would be designed and implemented by the software vendor. For example, Microsoft is a software vendor that sells the Windows OS. This OS has a heartbeat that assists a Windows instance (or installation) in checking for updates.

Service vendors are organizations that provide services. The distinction between software ven-dors and service venven-dors is that service venven-dors do not necessarily design associated software.

Instead, service vendors offer a service that can be checked periodically, thus producing a heart-beat. For example, an RSS feed created by a third party may periodically check Twitter for status updates from a certain set of users. At the U of C, heartbeats of this variety were generated by a Linux OS mirror that was periodically checked for updates by external organizations.

Managed hosting services provide software and service vendors with infrastructure. They can

provide different levels of service ranging from “bare metal”, which provides only a physical machine and nothing else, to fully administered databases and software suites. Thus, heartbeats directed at managed hosting services have nothing to do with the managed host, but have more to do with the service hosted there. As a consequence, managed hosting services can generate or receive heartbeats from a wide range of applications.

CDNs are similar to managed hosting services in that they host content for other organizations.

Thus, they may produce heartbeats related to their customer organizations. However, CDNs must also check the state of their network and ensure that updates are propagated in a timely fashion.

Thus, they can produce heartbeats of their own as well.

Figure 5.9 shows that each vendor has a distinct protocol/port usage profile. These differences reflect the diversity of services provided by each vendor. As with Figure 5.8, hashed areas indicate dead heartbeats.

Figure 5.9: Vendor-based analysis of heartbeats.

Microsoft is a software/service vendor whose heartbeats use TCP almost exclusively. Microsoft

443 were related to Skype traffic, Web application hosting, software updates, and user information collection. Heartbeats over TCP/993 were for Microsoft’s Outlook e-mail service. Heartbeats on user/dynamic port ranges were used by Skype to communicate with hosts on our network every 60 seconds.

The large number of dead heartbeats are caused by vendor-to-host communication. When Microsoft attempts to contact hosts on our network, there may be complications such as changes in IP address or the powering down of user machines. These heartbeats are inbound and make up 60% of all Microsoft heartbeats. The large proportion of dead heartbeats in this range indicate that Microsoft often attempts to establish heartbeats with transient hosts.

Google heartbeats relate to specific services. Heartbeats in the system port range for both TCP and UDP were concentrated on port 443 and directed towards Google’s servers. TCP heartbeats in the user/dynamic port range were directed towards our network, and provided Google’s services to hosts on our network. One example of these services was Google Hangouts.

Unlike Microsoft, Google produces very few dead heartbeats. This is somewhat surprising, since Google sends heartbeats to hosts on our network in a similar manner to Microsoft. Like Microsoft, the majority (55%) of Google heartbeats are inbound. However, there are not nearly as many dead heartbeats. Overall, the lifespan of Google’s heartbeats in this category are significantly shorter than Microsoft’s. Thus, it is likely that Google only sends heartbeats to known hosts over a short period of time. This approach is less likely to encounter the conditions that cause dead heartbeats.

Amazon exemplifies heartbeats for managed services. These heartbeats primarily used TCP ports 80 and 443. Inspection of the HTTP logs revealed that these heartbeats were not related to Amazon’s hosting services themselves. Rather, the heartbeats were generated by the wide variety of applications and Web pages that were being hosted on the servers. There were also some ICMP echo requests directed to Amazon’s servers.

Amazon also produces few dead heartbeats. Unlike Google and Microsoft, Amazon primarily

generates outbound heartbeats, which make up 95% of all Amazon heartbeats. Thus, there are no dead heartbeats directed to hosts internal to our network. Those that are dead are typically attempting to contact services that were hosted on an Amazon server, but have since changed their IP or hostname or do not exist anymore.

The Akamai heartbeats had several distinct patterns. The ICMP heartbeats were other Akamai hosts making ICMP echo requests to the Akamai node, and vice versa. The heartbeats in the user/dynamic port ranges for both UDP and TCP were used by Akamai for internal testing and reporting. TCP heartbeats in the system port range were all conducted over ports 80 and 443.

Many of the heartbeats conducted over port 80 were related to the Akamai Netsession Interface⁷, which periodically posts logs to Akamai servers.

All of the dead Akamai heartbeats are produced by our campus Akamai node. These dead heartbeats are used for the maintenance of the CDN. As such they are primarily outbound. Those in the TCP system port range are used for checking for updated content. These dead heartbeats indicate the content developer is currently down, or the IP address is stale. Those dead heartbeats in the UDP user/dynamic port ranges are used for internal testing of the CDN. The dead ICMP heartbeats are related to this as well.

Valve is a multimedia vendor that provides video games and game-related services to its cus-tomers. The heartbeats we observed were produced by Valve’s Steam video game client. These specific heartbeats occur with a period of 1.5-2 minutes on UDP/27,000-UDP/27,037. These port numbers⁸are used by Steam for game client traffic, game match-making, and in-home streaming.

Valve applications produce almost no dead heartbeats. This is because all heartbeats produced go to Valve’s servers. Due to this 98% of Valve heartbeats are outbound. Since these machines are always on there is very little churn in the destination IP addresses.

Overall, we see that dead heartbeats are much more common in P2P applications than non-P2P applications. In P2P applications, this is caused by churn. Those non-P2P applications that do

7https://www.akamai.com/us/en/products/mediadelivery/netsession-interface-faq.jsp

8https://support.steampowered.com/kb_article.php?ref=8571-GLVN-8711

produce a large number of dead heartbeats suffer from a similar problem. That is, they attempt to contact clients that are no longer able to respond. The lack of dead heartbeats related to Sality and Valve indicates that it makes use of always on servers that are resistant to churn, and as a consequence almost all heartbeats are alive.

5.5 Summary

In this chapter, we have explored the overarching characteristics of network heartbeats. We first discussed how network heartbeats make use of common periods, protocols, and ports and how this knowledge can be leveraged to identify applications. We then characterized the lifespans and periods of network heartbeats. Next, we investigated how heartbeat behaviour differs on various subnets. Finally, we analyzed how the heartbeats on different application architectures compare to one another.

Chapter 6 Discussion

In this chapter, we discuss the implications of our work. In Section 6.1, we review our objectives as they relate to our results. Section 6.2 highlights some of the challenges affiliated with network heartbeats. While discussing these challenges, we also speculate on how they may be overcome.

6.1 Objectives

At the beginning of this thesis, we posed three questions to guide our research. In this section, we review those questions. We discuss the answers based on our results in Chapters 4 and 5. In addition, we discuss the implications of our answers for network heartbeat research in general.

6.1.1 What are the general characteristics of network heartbeats on an edge network?

Through analyzing our results, we have developed a notion of the general characteristics of network heartbeats from a network operator’s perspective. They are summarized as follows:

• Heartbeats are generally unstable. In Section 4.2, we found that heartbeats are rarely produced in a continuous fashion as one might expect in a “perfect world”.

Instead, network heartbeats are far more likely to be produced for a limited time.

This is primarily due to irregularity, which is brought on by the prevalence of heart-beats in unmanaged environments. This is further amplified by the presence of P2P applications that produce lots of unstable heartbeats. Despite their rarity, sta-ble (regular) heartbeats are common in managed environments. Thus, they can be easily monitored for their presence or absence, which may indicate an issue.

When designing detection mechanisms, or investigating network heartbeats, one

confronted this issue, and so it is not known how detection methods fare against irregular heartbeats. Our own method was shown to have limited effectiveness in its original form, and required an adjustment to better detect them. However, these types of heartbeats are common and represent normal behaviour on an edge net-work. Thus, it is important to reliably detect and investigate them.

• In Section 5.2, we demonstrated that heartbeats are characterized by short lifespans and periods. This is brought on by both irregularity and design. In the case of ir-regularity, heartbeats may be longer from a user’s perspective. However, from the network level they appear to be short. This is due to the prevalence of heartbeats in unmanaged environments, similar to the instability discussed above. However, some applications, like Valve’s Steam video client, produce short heartbeats as a matter of course. The periods of heartbeats can also be designed to be short be-cause the application desires “fresh” information. In either case, observing heart-beats from the network level will result in the majority exhibiting short temporal characteristics.

These short periods and lifespans must be taken into account when investigating network heartbeats. The shortness of lifespan means that one cannot expect to observe all heartbeats on their network by sampling network traffic for a limited time. In addition, there is no guarantee that observing for a longer period of time will result in a longer heartbeat to analyze. As a consequence, detection methods must be effective on short-lived heartbeats in general. Likewise, detection methods must be effective on heartbeats with short periods.

• As discussed in Section 5.1, applications tend to produce heartbeats with a spe-cific period and port range. This was observed through analyzing the structural characteristics of heartbeats. Thus, popular applications will create clusters at the period and port range used. These clusters indicate applications or protocols that

are popular on an edge network.

This consistency means that the period and port can be used to classify heartbeats.

If one can map a particular port range and period to an application, one can monitor the use of these applications by analyzing the heartbeats. Adversaries intent on obfuscating their activities could use random ports to avoid classification, however the period may still form detectable structures in the heartbeat ecosystem. The period itself could also be altered, however this may make it difficult to coordinate communications. In addition, a great number of heartbeats on a particular port, but with differing periods, can be used to identify popular protocols or services even if these applications use end-to-end encryption.

• The analysis presented in Section 5.3 demonstrates that heartbeats are typically produced by end user applications. Our analysis of heartbeats in general revealed that most were produced by end user applications. The analysis conducted in Sec-tion 5.4 highlights that, typically, these applicaSec-tions are related to software/service vendors or P2P protocols. Our analysis of subnets revealed that unmanaged sub-nets produce significantly more heartbeats than managed. Unmanaged subsub-nets are representative of end user’s personal activities while the managed ones represent university services. This is supported further by subnets that contain NAT devices exhibiting similar heartbeat composition to unmanaged subnets.

From this, we can infer that heartbeats as a whole have little to do with important services. Most are simply a result of users “doing their thing” on the network.

However, this does not mean that heartbeats cannot be used to gain insight into network activities. A large number of regular heartbeats, for example, is likely to indicate some important service or machine. Likewise, abnormally large popula-tions of heartbeats related to an application can indicate issues with the health of a network, such as botnets.

6.1.2 What are the prevailing trends of the network heartbeat ecosystem on an edge network?

Our second research question is geared toward understanding how network heartbeats behave “in the wild”. We found that network heartbeats exhibited four key trends on our edge network:

• Section 4.5 shows that a large number of heartbeats are generated by P2P applica-tions. In our dataset, approximately 48% of all network heartbeats were generated by P2P applications. These heartbeats had a large impact on the heartbeat ecosys-tem of the network. P2P heartbeats were the primary reason why UDP was the prevalent protocol used for heartbeats. They also significantly contributed to the number of dead heartbeats observed. We found that this was because some P2P applications generate several heartbeats (to improve availability in the presence of continuous churn in the set of active peers), whereas other non-P2P applications typically generate only one.

We expect this trend to be common on edge networks like ours. P2P applications

We expect this trend to be common on edge networks like ours. P2P applications