Window Size Effects in Online Learning

The common observation window size for most of the results previously shown in this chapter is 10 seconds. This section investigates alternative window sizes and their affects on learning behaviour. The evaluation was conducted by running the entire NASA trace, with each request in the trace delivered in sequence, to the local emergent web server, using five different observation window sizes, from 1 second up to 20 seconds. The effects of this are shown in Fig. 5.19 and Fig. 5.20, with further extracted details shown in Table 5.1. In each experiment the web server is run for the same portion of the NASA trace. The evaluation shows the total number of adaptations that occur, and the number of exploration activities that are triggered. The analysis of the window size illustrated in Fig. 5.19 and Fig. 5.20 are rela- tively complex, with a set of interacting features. Overall the graphs show that the experiment using a 10 second observation window size completes the trace fastest, indicating that it had the best overall response times. There are two different rea- sons for this. First, the use of smaller observation windows causes over-reaction to the variations in the workload, and an unnecessarily large number of adaptations. These adaptations, whilst cheap, do momentarily impact the response time of the web server, resulting for example in the 5 second experiment having far higher peaks than the 10 second experiment shown in Fig. 5.20. The data for the 1 and 2 second window sizes, in Fig. 5.19, show even more volatility as the system attempts to over-fit to the environment. Second, the use of larger observation windows causes

1sec 2sec 5sec 10sec 20sec

# of classes 7 6 3 2 2

# of

adaptations (186)480 (124)376 (15)141 (20)104 84 (0)

Experiment

duration hour1 hour1 min50 min39 min48

Table 5.1: Adaptation details of observation window experiments, showing total number of adaptations performed, and adaptations performed under exploitation shown in brackets.

5.3. Additional Learning Aspects 137

under-reaction to the variations in the workload. This is exemplified by the 20 sec- ond observation window experiment, which has two exploration phases, but never performs any adaptations outside of those phases during exploitation, indicating that it has classified very broad environment ranges, missing important details.

0

50

100

150

200

250

300

1

162

323

484

645

806

967

₁₁₂₈

₁₂₈₉

₁₄₅₀

₁₆₁₁

₁₇₇₂

₁₉₃₃

₂₀₉₄

₂₂₅₅

₂₄₁₆

₂₅₇₇

₂₇₃₈

₂₈₉₉

₃₀₆₀

₃₂₂₁

₃₃₈₂

₃₅₄₃

R

es

po

ns

e

Time (ms)

Time (s)

1 sec 2 secs

Figure 5.19: Observation window experiments of 1 and 2 seconds.

0

20

40

60

80

100

120

140

160

180

1

137

273

409

545

681

817

953

₁₀₈₉

₁₂₂₅

₁₃₆₁

₁₄₉₇

₁₆₃₃

₁₇₆₉

₁₉₀₅

₂₀₄₁

₂₁₇₇

₂₃₁₃

₂₄₄₉

₂₅₈₅

₂₇₂₁

₂₈₅₇

₂₉₉₃

R

e

sp

o

n

se

Time

(

ms)

Time (s)

5 secs 10 secs 20 secs

Figure 5.20: Observation window experiments of 5, 10 and 20 seconds.

This demonstrates that the composition of the optimal observation window size is itself a challenge, and may have a different ideal setting across different types of application, different workload characteristics, and different hardware platforms. Beyond the static observation windows used here, it may also be useful in the future to explore dynamic observation window sizes.

5.4. Summary 138

5.4

Summary

This chapter presented two main scenarios that were used to evaluate the emergent system framework described in Chapter 4. The first scenario consists of evaluating the emergent framework to autonomously assemble and optimise a single instance of a web server. The second scenario consists of exploring the emergent framework in distributed settings. Both scenarios aim to evaluate learning strategies in terms of convergence rate (i.e. how fast it converges) and accuracy (i.e. if the system lo- cates the optimal composition). Furthermore, this chapter also evaluates additional aspects of the learning process, such as the environment classification algorithm and the size of the observation window used during the learning process.

In detail, for the local scenario, this chapter evaluates both the brute-force and feature-based approach applied to a single instance of the web server. The results show that the emergent system concept enables the continuous composition and optimisation of the web server according to the workloads. Furthermore, the results show that the feature-based approach converges, in the best-case scenario, 7 times faster than the brute-force, and more than 2 times faster in the worst-case scenario. In the distributed scenario, the chapter evaluates different coordination strategies for the learning process. The results show that the centralised altruistic approach converges to the optimal solution, but this approach is inefficient as the number of composition grow exponentially in distributed settings. The decentralised selfish strategy converges faster and accurately in some environments, but suffers from learning interference effects in other scenarios. Finally, for the coordination analysis, this chapter show that a hybrid coordination strategy (the hierarchical coordination) converges faster and accurately considering a larger search space, whilst avoiding interference effects and invalid global compositions. This approach, however, is only applicable in domains where the feature-based assumptions are maintained.

Finally, the chapter evaluates the environment classification algorithm, and shows the results of experimenting with different sizes for the observation window. The re- sults show that the environment classification suffers from classifications that present overlapping ranges, mid-exploration workload changes and when some important

5.4. Summary 139 environment feature is not defined as an event (perception data, see Chapter 4). The overlapping ranges classification confuses the system during exploitation phase, causing instability in the system performance. The mid-exploration changes ‘de- ceives’ the system to converge to composition that is not as good as the known optimal. The observation window size depends on the application domain and on the system’s workload, so that for different scenarios, different sizes might be better. The results for the observation window size in the explored scenario and workload points to the 10 second window as the most appropriate size for a more accurate learning. However, further investigation on this aspect of the system is necessary, including an investigation on dynamic size of the observation window.

Although the different aspects of the emergent framework evaluated in this chap- ter is not applicable to every application domain, nor covers every learning chal- lenge in Chapter 3, the presented solution implements important concepts that were demonstrated to be key to realise the paradigm of emergent systems. Furthermore, the shortcomings of the presented solution establish the next research steps that are detailed in the next chapter as future work.

CHAPTER

6

Conclusion

Autonomic computing has emerged as an important area to address the increasing complexity in developing, configuring, and managing software systems. Due to the uncertainty of highly dynamic operating environments, the development of such solutions becomes very challenging. As a response to the complexity of creating autonomous solutions, this thesis has introduced Emergent Software Systems, with a real-world implementation of the concept and an extensive empirical evaluation.

The main goal of Emergent Systems is to autonomously assemble and optimise software systems in both local and distributed environments with minimum human involvement. As a consequence of the presented concept, systems become able to accommodate changes in their operating environment with no offline training nor predefined domain information, reducing the engineering effort required to create autonomous solutions. Furthermore, the ability to freely explore the available design possibilities, with no restrictions, enable the system to handle unexpected changes. This chapter revisits previous chapters, highlighting this thesis’ main contribu- tions and the novelty of the presented approach. Moreover, this chapter revisits the research questions defined in Chapter 1, addressing the questions with references to the results presented in Chapter 5, and concepts discussed in both Chapter 3 and Chapter 4. Finally, this chapter describes several avenues for future work, and concludes the thesis with the final remarks.

6.1. Thesis Summary 141

6.1

Thesis Summary

Chapter 1 introduced the thesis, contextualising the work and its relevance to the

Autonomous Computing research area. This chapter also described this thesis’ main objectives and contributions, concluding with a presentation of the thesis’ structure.

Chapter 2 presented the background concept used in this thesis, whilst also pre-

senting relevant related work. This chapter presented a literature analysis, compar- ing the concept of Emergent Systems with key work in autonomous systems, and detailing the advantages of Emergent Systems and the gap this concept addresses.

Chapter 3 defined the concept of Emergent Software Systems in terms of au-

tonomously composing software systems from small units of behaviour and online learning techniques. This chapter also introduced the main challenges in realising Emergent Systems in both local and distributed scenarios.

Chapter 4 described the framework implementation that supports the concept of

Emergent Systems. This chapter introduced the Perception, Assembly and Learning (PAL) modules as the core framework to realise Emergent Systems in local settings. Furthermore, this chapter presented an extension to the PAL framework to support the concept of Emergent Systems in distributed scenarios.

Chapter 5 evaluated the emergent framework applied to datacentre software.

The results presented in this chapter demonstrate the feasibility and potential of Emergent Systems. Furthermore, this chapter also analysed the limitations of the presented solutions, presenting further challenges to be addressed in future work.

In document Emergent software systems (Page 136-141)