2.4 Experimental Details
2.4.4 Impedance Spectroscopy Based Measurements with
All regional and specialist agents are deployed on Planetlab nodes. The service direc-tory and knowledge agent run on a machine at MIT. User agents in this implementation operate as Firefox extensions inside web browsers on end user machines. Implementing user agents as Firefox extensions has a number of advantages. Firstly, Firefox supports multiple platforms, including Windows, MacOS, and Linux. Secondly, Firefox has an ac-tive community of extension developers. Thirdly, the Firefox addons web site5 provides a convenient distribution channel to reach a large number of users around the world. For platform independence, I implemented user agents using Javascript. User agents do not advertise any services; they only issue notifications and diagnosis requests.
All other agents are implemented as Python 2.4 scripts running as Apache mod python handlers interfacing with a MySQL database backend. There are several advantages to implementing agents using Apache, Python, and MySQL.
Firstly, Apache is multithreaded, allowing agents to handle multiple requests simultane-ously. Secondly, Apache can demultiplex requests to multiple diagnostic agents operating as HTTP request handlers on the same web server. Thirdly, Apache provides a common logging infrastructure. Fourthly, Apache provides tools for monitoring HTTP server sta-tus to assist in debugging. A disadvantage of Apache is the additional memory and CPU overhead of running a web server.
Python has several advantages as well. Firstly, as a high-level scripting language that provides many built-in features and modules, it allows for much more rapid development than C or Java. Secondly, Python is a popular and easy to understand language, which makes it easier for others to build their own agents using the code I developed. A disadvan-tage of Python is that it is an interpreted language, and is slower and uses more memory than a natively compiled program written in C or C++. This disadvantage is somewhat mitigated by the fact that Python does support modules written in C. I decided to accept this tradeoff because my experiments primarily evaluate the functionality of CAPRI, and not the speed of diagnosis.
I choose to use a MySQL database rather than memory or disk files for storing data in
5https://addons.mozilla.org/en-US/firefox
this implementation for several reasons. Firstly, a database provides atomicity and consis-tency to support multithreaded operation. Secondly, a database provides a standard SQL interface for querying and aggregating information agents collect. Each Python agent main-tains its component information base, knowledge base, service table, and component class and property definitions in database tables. In addition, web server history agents, stats agents, and knowledge agents record and aggregate notifications they receive using con-nection history and diagnosis history tables.
Agents send and receive service advertisements using a centralized agent directory server located at MIT. All agents know the URI of the service directory. In order to dis-cover newly available services, Python agents reload their list of available services from the agent directory approximately once an hour. To take advantage of new regional agents that become available, user agents reload available services approximately once a day. Re-gional and specialist agents use requester AS restrictions and costs to preferentially handle requests from requesters in the same AS. Thus users who are in the same AS as a regional agent send diagnostic requests and notifications to that regional agent. Similarly, regional agents request beliefs from specialist agents within their AS if available.
To support the creation of new diagnostic agents, most of the common functionality of CAPRI agents such parsing messages, maintaining a component ontology, managing a ser-vice table, constructing failure dependency graphs, and performing probabilistic inference is contained in a Python module. Python agents use the OpenBayes6 module to perform probabilistic inference and use the PyXML module7to parse XML messages. Knowledge agents use the SMILE reasoning engine to learn probabilistic dependencies.8 Creating a new diagnostic agent simply involves extending the default diagnostic agent implementa-tion with the desired capabilities and placing new component class and property definiimplementa-tions and service descriptions in the appropriate locations.
Below are tables of the notification subscriptions and the services offered by diagnostic agents in my prototype implementation. Input and requester restrictions are not shown here. In addition, most services also have a corresponding “local” version with lower cost for those requesters in the same AS as the agent. Appendix B contains the full service descriptions.
6http://www.openbayes.org/
7http://pyxml.sourceforge.net/
8SMILE is available from the Decision Systems Laboratory, University of Pittsburgh (http://dsl.sis.pitt.edu)
Service ID Input Output Cost
diag:http HTTP Connection Explanations 1000
status localNet.status
dnsLookup.hostname httpServer.status localNet.srcIP dnsLookup.status ipRouting.destIP ipRouting.status
obs:stats HTTP Connection Observation 1000
destHash (required) users avgLatency lastSuccess lastFailure recentStatusDist
knowledge:http Knowledge 100
Local Network.status
Local Network.consecFailuresFromUser HTTP Connection.status
DNS Lookup.status IP Routing.status HTTP Server.status
HTTP Server.consecFailuresToServer Outbound Conn Test.probeResult Firefox Error Test.ffoxErrorCode Table 8.1: Regional agent services
Agent/Service ID Input Output Cost
Web Server History HTTP Server Belief 1000
Agent ipAddr (required) status
bel:webserver.status connTime (required) Observation
hostHash consecFailuresToServer
httpConn Knowledge
httpConn.status HTTP Server.status httpConn.destHash
httpConn.destIP httpConn.srcIP httpConn.connTime httpConn.elapsedTime httpConn.ffoxErrorTest
httpConn.ffoxErrorTest.-ffoxErrorCode
Web Server History HTTP Server Observation 1000
Agent ipAddr (required) consecFailuresToServer
obs:webserver.cfts connTime (required) totalConsecFailuresToServer
DNS Lookup Agent DNS Lookup Belief 10000
bel:dnslookup.status hostname (required) status Observation
verifyDNSLookupTest.-dnsLookupResult Knowledge
DNS Lookup.status
AS Path Agent IP Routing Belief 100000
bel:iprouting.status srcIP (required) status destIP (required) Observation
asPathTest.asPathTestResult Knowledge
IP Routing.status
Stats Agent HTTP Connection Observation 100
obs:stats destHash (required) users
avgLatency lastSuccess lastFailure recentStatusDist Table 8.2: Specialist agent services
Service ID Input Output Cost
knowledge:http(learned) Knowledge 1000
Local Network.status
Local Network.consecFailuresFromUser HTTP Connection.status
DNS Lookup.status IP Routing.status HTTP Server.status
HTTP Server.consecFailuresToServer Outbound Conn Test.probeResult Firefox Error Test.ffoxErrorCode
Verify DNS Lookup Test.dnsLookupResult AS Path.status
AS Path Test.asPathTestResult AS Hop.status
AS Hop Test.asHopTestResult
knowledge:specialist Knowledge 1000
Local Network.status
Local Network.consecFailuresFromUser DNS Lookup.status
IP Routing.status HTTP Server.status
HTTP Server.consecFailuresToServer Outbound Conn Test.probeResult
Verify DNS Lookup Test.dnsLookupResult AS Path.status
AS Path Test.asPathTestResult AS Hop.status
AS Hop Test.asHopTestResult Table 8.3: Knowledge agent services
Service ID Input
notify:connHist HTTP Connection status
destHash srcIP destIP connTime elapsedTime ffoxErrorTest
ffoxErrorTest.ffoxErrorCode notify:diagHist HTTP Connection
localNet localNet.status localNet.ipAddr
localNet.consecFailuresFromUser localNet.totalConsecFailuresFromUser httpServer
httpServer.status
httpServer.consecFailuresToServer dnsLookup
dnsLookup.status dnsLookup.hostname ipRouting
ipRouting.status ipRouting.srcIP ipRouting.destIP outboundConnTest
outboundConnTest.probeResult outboundConnTest.probeURI ffoxErrorTest
ffoxErrorTest.ffoxErrorCode
Table 8.4: Regional agent notification subscriptions
Agent/Service ID Input
Stats Agent HTTP Connection
notify:connHist(destHash) status
destHash (required) srcIP
destIP connTime elapsedTime ffoxErrorTest
ffoxErrorTest.ffoxErrorCode Web Server History Agent HTTP Connection
notify:connHist(destIP) status
destIP (required) srcIP
connTime elapsedTime ffoxErrorTest
ffoxErrorTest.ffoxErrorCode Table 8.5: Specialist agent notification subscriptions
Service ID Input
notify:diagHist HTTP Connection localNet (required) localNet.status (required) localNet.ipAddr (required)
localNet.consecFailuresFromUser (required) localNet.totalConsecFailuresFromUser httpServer (required)
httpServer.status
httpServer.consecFailuresToServer dnsLookup (required)
dnsLookup.status dnsLookup.hostname ipRouting (required) ipRouting.status ipRouting.srcIP ipRouting.destIP
outboundConnTest (required) outboundConnTest.probeResult outboundConnTest.probeURI ffoxErrorTest (required) ffoxErrorTest.ffoxErrorCode
Table 8.6: Knowledge agent notification subscriptions
Chapter 9
Experimental Results
The purpose of the CAPRI architecture is to provide a common framework for distributed diagnosis among heterogeneous diagnostic agents. This chapter evaluates the effective-ness of distributed diagnosis using the diagnostic agents in the prototype implementation described in Chapter 8. I demonstrate that multiple heterogeneous diagnostic agents with different capabilities and located in different parts of the network can use CAPRI to effec-tively diagnose real-world HTTP connection failures in a distributed manner in real time.
Unlike previous research in distributed diagnosis [44, 67, 98], agents in CAPRI can in-corporate information from multiple types of diagnostic tests, including both active and passive measurements, and can deal with incomplete information caused by network fail-ures. In addition, I demonstrate the ability to add new diagnostic agents to the system and show that existing diagnostic agents can take advantage of new dependency knowledge and new diagnostic agents to improve the accuracy and cost of diagnosis. This chapter also illustrates the effectiveness of aggregation and caching to reduce the cost of diagnosing multiple similar failures for scalability. Unlike previous research that only considers the probing costs of single diagnoses [76], this experiment shows how agents can manage both the probing and communication costs of multiple, repeated diagnoses.
This chapter presents experimental results demonstrating the advantages of the CAPRI architecture. I show that regional agents dynamically select diagnostic actions without domain-specific knowledge. Heterogeneous diagnostic agents use belief propagation to di-agnose failures in a distributed way even when no single agent has enough information to adequately diagnose a failure. I also find that caching and probabilistic inference reduces the number of diagnostic requests sent to specialists and to regional agents. Also, I show that adding learned dependency knowledge improves the accuracy of diagnosis while re-ducing cost. In addition, adding a new type of diagnostic agent can reduce the response time of certain requests. I also show that the confidence metric produced in a diagnosis can be used to predict the expected accuracy of a diagnosis if the dependency knowledge used for diagnosis is correct.
The experimental evaluation described in this thesis differs from previous work in dis-tributed Internet fault diagnosis in that it evaluates the performance of real-time, online distributed diagnosis for real-world failures. Previous researchers generally focus on off-line diagnosis or diagnosis of simulated failures in simulated networks. Evaluation using world failures can provide better insight into the challenges of diagnosis under
real-world conditions, in which network failures can interfere with diagnostic communication and diagnostic tests may produce noisy or inaccurate results. The data for the experimen-tal results shown below were collected over a period of over one month, from March 6 to April 10, 2007. In this experiment, diagnostic agents collect information about a total of approximately 500,000 HTTP connections per day, and diagnose approximately 3,000 HTTP connection failures per day.
9.1 Distributed diagnosis
Chapter 8 describes the range of diagnostic agents that I implemented and deployed. Figure 9-1 illustrates the knowledge that agents use to diagnose failures in my implementation.
Each type of diagnostic agent knows a different subset of this dependency knowledge. The CAPRI architecture enables distributed agents to discover one another and communicate observations, beliefs, and knowledge to perform distributed fault diagnosis.
HTTP Conn status Local Net
status
Web Server
status DNS Lookup status
IP Routing status Consec failures
to server
Consec failures from user
DNS Lookup Test result
AS Path Test result
Outbound Conn Test result
AS Path status
Firefox error code User Agent Knowledge Web Server
History Agent Knowledge
DNS Lookup Agent Knowledge
AS Path Agent Knowledge
Figure 9-1: Agents with different capabilities and knowledge exchange observations and beliefs to diagnose failures.