Distributed Emergent Systems Challenges

This section expands on the intrinsic challenges presented in the previous section, focusing on the emergent systems’ challenges and properties that stand out consid- ering a distributed scenario. These challenges are: the combinatorial explosion in the search space, the locus and personality control of meta-structures, information sharing, interference effects and behavioural mismatch.

Combinatorial explosion in search space

The presented Emergent Software vision is fundamentally based on a combinatorial learning process. These systems are built out of the process of combining smaller

3.2. Emergent Software Systems Challenges 65 components into architectures and testing these architectures at runtime. As com- ponents are added to the repository the number of possible architectural assemblies increase exponentially. This scenario becomes considerably worse when consider- ing distributed systems, where the global system architecture is the result of the combination of a set of micro-architectures. Considering that the addition of one component to the repository exponentially increases the number of possible micro- architectures, in the distributed scenario this property has a cascading effect when considering the quantity of participating nodes in the system. The global system ar- chitecture is the overall combination of all participating node’s micro-architectures. To illustrate how quick the search space grows in a distributed scenario, consider the following example: Two identical web servers with 50 compositions each running on two distinct nodes, and one load balancer with 10 valid compositions running on another node. The load balancer is responsible to forward requests to both web servers. This system has 25,000 valid compositions (50 times 50 times 10). If we add another web server node with 50 composition, the number of global compositions increases to 1,250,000 (50 times 50 times 50 times 10). This is a inherent property of emergent systems and an essential problem to be addressed in order to show the feasibility and to guarantee the consolidation of the emergent system paradigm.

The combinatorial explosion problem is an open issue, and the author does not claim to provide the ultimate solution for this problem. However, this thesis shows how to explore some properties and characteristics of some application domains to mitigate the combinatorial explosion innate of emergent systems. For more details on the proposed solutions, please refer to Chapter 4.

Locus and personality of control

The meta-structures of emergent systems is responsible to i) compose system using software units, ii) collect perception data generated by the components at runtime and iii) learn about the collected information. These meta-structures can operate in different locations with different ‘personality’. The meta-structures can operate controlling arbitrary groups of the nodes in the system. They can control the op- erating of individual nodes in a complete decentralised fashion, or clusters of two,

3.2. Emergent Software Systems Challenges 66 three or four nodes, etc., up to all nodes of the system, maintaining a centralised system’s control. Additionally to that, meta-structures can adopt different person- ality by, for example, acting in an entirely selfish manner according to the groups local interests, ignoring the rest of the system, or by acting in an altruistic way making local decisions to benefit the global system’s interests.

Interference effects

Interacting nodes in a distributed emergent system might interfere with each other’s perception of the world, influencing both their metrics and events. Metrics are influenced from interacting nodes when there is a functionality dependency between nodes. For instance, when a load balancer running on a node receives a request, it forwards the request to a web server running on another node to process it. In this scenario, the load balancer’s response time depends on the time the web server takes to handle the request. If the web server is in a sub-optimal composition, it directly affects the load balancer’s response time, even if the load balancer is in an optimal composition. The same happens to events. For instance, if events are registering information about the requests (e.g. type of files requested – image, text, video, etc.), the perception of the operating environment of the web server is determined by the files the load balancer is forwarding to the web server. In case the load balancer changes its scheduling policy, the web server perceive a change in the operating condition. These interference effects, in a large distributed environment, can compromise the quality of the learning process, making the system less efficient.

Information sharing

Information sharing is essential to reach consensus in a distributed scenario and to optimise the exploration phase in the learning process. Sharing information among groups of nodes controlled by meta-structures gives the group an external perceptive of the system and how the group is effecting the system’s goal. Furthermore, infor- mation sharing can help optimise the learning process by diving architectural explo- ration among nodes with similar resources and share the outcome among them. This mitigates the combinatorial explosion problem by diving the search space among

3.3. Thesis Scope 67