Next, we summarize the works that allow for runtime analysis of distributed systems.
6.2.1. System Monitoring
There are a number of research works that monitors the execution of the distributed sys-
tem. The monitoring allows the system developer to understand the key behavior of the
distributed system, and to diagnose the problem in a more effective way. For example,
Gunter et al. [41] proposes to dynamically monitor the distributed system by instrumenting
network devices in a light-weighted manner. Chen et al. [24] records runtime paths that
requests follow in a distributed system to manage failure and evolution of the system. P2
Monitor [77] develops an application logging, monitoring and debugging facility built on
top of P2 system. D3S [52] allows the developers to specify prediate over the distributed
system, and would output a sequence of state changes when failure occurs.
6.2.2. Message Logging
Message logging is a well-studied area for analyzing distributed systems. Aguilera et al. [2]
uses message-level traces to debug performance in distributed systems with black boxes.
Netlogger [80] records the event logs in a distributed system to provide detailed end-to-
end application and system-level monitoring. Pip [72] proposes infrastructure that allows
the user to compare his/her expected behavior of the distributed system with the actual
behavior that is recorded through instrumentation of the system. Liblog [37] records log
execution for C/C++ applications in a distributed system, and deterministically replay the
logs to help debug the system. Google has also proposed Dapper [76], which is a tracing
infrastructure that help debug the distributed system. X-trace [34] designs a tracing frame-
the deterministic replay with symbolic debugging. WiDS Checker [53] uses simulation to
replay the execution of the distributed system, and allows the user to specify high-level
properties which are fed to the checker for verification.
6.2.3. Network Provenance
Network provenance is an area which draws the attention of researchers recently. Compared
to message logging, network provenance explicitly captures the causality relationship among
events in the distributed system, providing a more convenient tool for the user to debug
the system.
Network provenance has been proposed and developed by ExSPAN [95] and DTaP [94].
These two proposals apply the concept of data provenance to the networking field to sup-
port network diagnosis and forensics. In Chapter 5, we adopted ExSPAN’s provenance tree
model as a starting point, and developed compression schemes based on the model. Secure
Network Provenance [92] enables network provenance in an adversarial environment, where
misbehaving nodes can be detected even if they lie. Negative provenance [85] extends the
network provenance to explain the missing event in the network. There are a number of
works that have already used network provenance to systematically debug the system. Dif-
ferential provenance [16] debugs the distributed system by identifying the difference between
a faulty provenance tree and a reference provenance tree. Meta Provenance [84] proposes
to debug the software of network devices with the help of provenance. Our compression
scheme is expected to work on all the systems that support network provenance.
6.2.4. Provenance Compression
In database literature, a number of works have considered optimization of provenance stor-
age. However, we differ significantly in our design due to the distributed nature of our
target environment. We briefly list a few representative bodies of work, and explain our
Woodruff et al. [83] reduce storage usage for maintaining fine-grained lineage (i.e., prove-
nance) by computing provenance information dynamically during query time through in-
vertible functions. Their approach makes tradeoff between storage efficiency and accuracy
of provenance. On the other hand, our approach requires no such tradeoff, achieving the
same level of accuracy as queries on uncompressed provenance trees.
Chapman et al.[15] develop a set of factorization algorithms to compress workflow prove-
nance. Their proposal does not consider a distributed setting. For example, node-level
factorization (combining identical nodes) requires additional states to be maintained and
propagated from node to node during provenance maintenance to resolve potential ambi-
guities. Maintaining and propagating these states can lead to significant communication
overhead in a distributed environment. In contrast, our solution uses equivalence keys to
avoid comparing provenance trees on a node-by-node basis, and hence minimizes commu-
nication overhead during provenance maintenance.
Our compression technique implicitly factorizes provenance trees at runtime before removing
redundant factors among trees in the same equivalence class. Olteanuet al.[65][66] propose
factorization of provenance polynomials for conjunctive queries with a new data structure
called factorization tree. Polynomial factorization in [66] can be viewed as a more general
form of the factorization used in the equivalence-based compression proposed in Chapter 5.
If we encode the provenance trees of each packet as polynomials, the general factorization
algorithm in [66], with specialized factorization tree, would produce the same factorization
result in our setting. Our approach is slightly more efficient, as we can skip the factorization
step by directly using equivalence keys at runtime to group provenance trees for compression.
Exploring the more general form of factorization in [66] for provenance of distributed queries
is an interesting avenue of future work.
ProQL [46] proposes to save the storage of single provenance tree by (1) using primary
keys to represent tuples in the provenance, and (2) maintaining one copy for attributes of
online compression algorithm to further reduce storage. ProQL does not consider storage
sharing across provenance trees. Amsterdamer et al. [5] theoretically defines the concept
of core provenance, which represents derivation shared by multiple equivalent queries. In
our scenario, the shared provenance tree of each equivalence class can be viewed as core
provenance.
Xie et al. [86] propose to compress provenance graphs with a hybrid approach combining
Web graph compression and dictionary encoding. Zhifeng et al. [10] proposes rule-based
provenance compression scheme. Their approaches on a high level compresses provenance
trees to reduce redundant storage. However, these approaches require knowledge of the
entire trees prior to compression, which is not practical, if not impossible, for distributed
CHAPTER 7
Future Work
In addition to the work that has been done, we propose future work that aims at providing
a more complete framework that allows for unified static analysis and runtime analysis of
declarative distributed systems. The future work is mainly composed of two parts:
• A more complete framework for provenance compression. To make our frame-
work more complete, we intend to add more features into our current framework, along
with more evaluation on additional applications.
• Optimization of static analysis. We plan to optimize static analysis by designing
algorithms that could incrementally maintain derivation instances for evolving NDlog
programs.
7.1. A More Complete Framework of Provenance Compression
Though our provenance compression scheme has already achieved significant storage reduc-
tion for packet forwarding and DNS resolution applications, we try to make the framework
more complete by (1) allowing the user to dynamically specify the equivalence relation of
provenance trees, and (2) performing more evaluation over the prototype we have now. We
introduce these two tasks in more detail below.
7.1.1. Dynamic Definition of Equivalence Relation
When we perform the provenance compression now, the equivalence relation for provenance
trees is pre-defined. This definition, though effective in reducing storage space, could still
incur large storage overhead under certain circumstances. For example, consider a slow-
changing table T (e.g., a routing table). Two provenance trees that use different tuples in
current definition of the equivalence relation. However, if we excludeT from the definition
of the equivalence relation, the above two provenance trees can be viewed as equivalent and
got compressed.
Therefore, we intend to support such dynamic definition of the equivalence relation, so that
users could specify their own equivalence relations that mostly meet their needs. We expect
the new design to further compress provenance trees that differ in a portion ofprovenance
nodes, rather than only one singleprovenance node.
The change of design could possibly make provenance maintenance and query more compli-
cated, as it now requires nodes in the distributed system to selectively maintain provenance
information – i.e., these nodes need to compress provenance nodes involved in the definition
of the equivalence relation, but not those not involved. We expect to explore the new design
more carefully in the future.
We also plan to evaluate the storage overhead using the dynamic definition of the equiv-
alence relation, and compare it with the static definition, over a few popular networking
applications, such as packet forwarding.
7.1.2. Provenance Maintenance for SDN Applications
The two applications we are evaluating now – i.e., packet forwarding and DNS resolution
– are relatively simple, in that their specification involves no more than five NDlog rules.
To show the generality of our solution, we plan to apply the compressed provenance to
applications in software-defined networking, such as Ethernet MAC learning or Ethernet
address resolution. A typical SDN-enabled distributed system involves multiple types of
network devices, including a controller, a number of bare-metal switches (e.g., Openflow
switches) and end hosts. Each network device will be modeled in NDlog rules and the
provenance information will be collected and compressed for packets traversing the network,
We will not only apply our solution to the selected SDN application, but also redo all the
evaluation for the new application. This includes measurement of the provenance storage
(growth), the bandwidth utilization and the query latency.
7.1.3. A Cost Model for Selection between Alternative Solutions
The experimental results In Section 5.6.1 show that the reactive maintenance strategy of
DTaP [94] could save more storage than our online compression scheme, sacrificing a certain
amount of time during querying. As a result, the user needs to make tradeoff in reality when
selecting either solution. A rigorous cost model could ease the pain of making a decision as
to which strategy to take. The cost model is expected to take into account several factors:
(1) frequency of querying, where the online compression scheme is favored when the query
frequency is high; (2) storage availability, where DTaP is preferred when storage is scarce.
(3) bandwidth availability, where DTaP consumes less bandwidth, and (4) cost of replay,
where the online compression scheme is a winner if replay is costly – e.g., no duplication
system exists.