Introduction to complexity concepts

There is no common definition of the term “complexity”. The Oxford dictionary defines complexity as something “made of usually several closely connected parts”. The

tel-00481367, version 1 - 6 May 2010

Latin “Complexus” signifies "entwined", "twisted together”. ”Complicated” uses Latin ending “plic” that means “to fold”. “Complex” uses “plex” that means “to weave”.

Many attempts had been made to develop a generalized understanding of complexity, and, ultimately, a Theory of Complexity - a systems theory (Lucas 2000) that consists of many interacting components and many hierarchical layers. Note that a system is called complex if it is impossible to reduce its overall behaviour to a set of properties characterizing its individual components (Lucas 2000); interactions at collective level in such system can produce properties that are simply not present when the components are considered individually.

The quantification (measurement) of the systems’ complexity, including complexity of classification process, plays the key role in T-DTS as it is responsible for optimizing tree structure of Neural Networks. This motivated us to find a new approach for the analysis of complexity and synthesis of complex systems.

We highlight in this chapter our general interest in investigating a complexity phenomenon and stress the need in a clearly visible and solid classification complexity estimator, to the extent possible while using our ad hoc approach described below. We begin by considering the theoretical aspects of the complexity concept. Considering the given aggregation, this concept can be stratified from the simplest level to the most complex:

1. Static complexity: the simplest form of complexity that relates to static systems and is generally studied by scientists using developed mathematic tools (Lucas 2000). For example this form of complexity is studied by such techniques as Algorithmic Information Theory (Chaitin 2005).

2. Dynamic complexity: extends upon static complexity by adding the dimension of time that can improve or worsen the static situation (Wolfram 1994). Given interest in experimental repeatability in science, is to observe dynamic and measure this type of complexity (Lucas 2000) “which is a one way to exhibit the phenomenon complexity because complexity as phenomena is born not statically, but dynamically (Appendix B).

3. Evolving complexity: relates to systems that evolve through time into different systems (Wolfram 1994); the best known class of this phenomenon is usually described as organic: open ended mutation. As all such systems are unique, there are symmetries present in the arrangements that would allow one to measure these systems. For example, it is possible to analyze the complexity of an evolving system from an evolutionary viewpoint as a set of specific already investigated parts or patterns (e.g. DNA code) that

tel-00481367, version 1 - 6 May 2010

can also have numerous combinations that have not yet occurred and thus have not been studied (Lucas 2000).

4. Complexity of self-organizing systems: this form of complexity is based on the idea of comprising. According to Lucas (Lucas 2000), this is the self-maintaining type of systems that operating at the edge of chaos, aggregate in nonlinear ways the structures and complex mix of types 1-3 above mentioned (Wolfram 1994).

Let us mention that the complexity concept categorizing on 1-4 is performed by Lucas (Lucas 2000) in very informal way, but it is not the fault of author. Thus, even the simplest Static type of complexity that employing precise mathematic tools does not have the common definition (Saakian 2004). Although, this work (Lucas 2000) is useful to introduce a common context of the phenomenon complexity. Therefore, before applying a quantitative technique (meaning we concern with Static type of complexity) of estimating complexity, we need to decide whether they are, in fact, complex in any of the senses mentioned above (Bak 1996). However, the problem with deciding and determining complexity is typically done in an informal form of comparison. It means that the whole spectrum of complexity assessment statements will be in the form of "x system … is more complex than y system" (Edmonds 1999). Also, there is some point of a transition from

“simple” to “complex”; the assumed nature of this point further complicates the formalization of complexity estimations.

Therefore, we will provide the pragmatic solution for describing categories of complexity. The philosophical issue related with complexity will henceforth be ignored.

To classify different types of complexity, we define the following criteria that we later use to organize concepts in several groups:

• Criterion of size: for example, the size of a genome, the number of species in a biosphere. Size could be an indicator of difficulty, but for strong definition of complexity, such criterion is inter-related.

• Minimum description length criterion: is based on Kolmogorov’s idea of complexity based on the minimum possible length of a description in some language (usually that of a Turing machine) (Shalizi 2005), (Chaitin 2005). We should discuss this criterion in details later in this section.

• Criterion of variety: variety of basic components of a concept. Variety is the key point of evolutionary processes. For example, human teeth including its organization and functions are more complex than shark teeth regardless the quantity (Edmonds 1999), (Lucas 2000). Variety is the necessary feature of complexity but is not sufficient for it.

tel-00481367, version 1 - 6 May 2010

• (Dis)Order: Complexity is a mid-point between order and disorder (in board meaning of these terms) (Permana 2003).

There are certain difficulties in applying the listed criteria to building a common solid complexity hierarchy; most notable, the criteria originate from absolutely different fields and these origins cannot be ignored. Thus, we are going to rely on a heuristic inventory done at the Horgan’s work (Horgan 1995). In his survey, Horgan analyses more than 30 different ways of categorizing complexities; in this work, we will consider the four of them: algorithmic complexity, computational complexity, entropic complexity, grammatical complexity.

In the next section we describe the most important of the four considered complexity categories, the computational complexity. Based on structure and complexity of a given data, we propose a combined context-dependent measure of complexity of associated computations. We describe our contribution of a novel classification complexity estimation technique named ANN-structure based classification complexity estimator; we explain its connections to other complexity estimation approaches according based on proposed classification complexity measurement hierarchy.