Optimizing problems with qualitative variables, that is, variables with discrete attributes, states, or values, is inherently different from optimi- zation of quantitative or numerical problems. Here the problem is one of arranging the elements to minimize or maximize a result. In some cases we might want to eliminate some elements as well, so that the number of things to be rearranged is itself part of the problem. Combinatorial opti- mization is particularly addressed within the field of traditional artificial intelligence, especially often where the task is to arrange some proposi- tions in such a way that they conform to some rules of logic. “Production rules,” if-then statements such as “ifAis true then doC,” are evaluated and executed in series, to obtain a valid logical conclusion or to deter- mine the sequence of steps that should be followed in order to accom- plish some task. Arranging the order of the propositions is a combina- torial optimization problem.
The statement of the goal of an optimization problem is theobjective
function, sometimes called a “cost function.” For example, consider a
classic combinatorial problem called the traveling salesman problem (TSP). In this problem, a map is given with some number of cities on it, and the objective function to minimize is the distance of a route that goes through all the cities exactly once, ending up where it started.
Thesimple permutations of some elements are all the possible order-
ings of an entire set. In a very small traveling salesman problem, we might have three cities, called A, B, and C, and their simple permutations would be ABC, ACD, BAC, BCA, CAB, and CBA. Many combinatorial
optimization problems require finding the best simple permutation of some objects. There arenfactorial (written asn!) simple permutations of nelements. Thus with our example of three things, there are 1×2×3= 6 permutations. The point to realize here is that this number, which rep- resents the size of the search space or set of possible solutions to the prob- lem, gets big quickly. For instance, with only 10 elements the search space contains 3,628,800 possible simple permutations. Any problem that is likely to exist in the real world, for instance, a complicated sched- uling or routing problem or cognitive model, will almost certainly have more elements in it than that; we need to find ways to search for an an- swer without trying every possible solution. Optimal allocation of trials, again.
Simple permutation problems are often represented, conceptually at least, as trees. The three-element problem is just too trivial, so let’s con- sider a simple permutation problem with four elements (see Figure 2.13). The first decision is where to start, and it has four options, A, B, C, or D. Once we’ve selected one of these, say, A, then we are faced with the next decision: B, C, or D? Each step of the process reduces the number of op- tions by one. Say we choose B, now the decision is between C and D. Se- lecting C, we have no real choice, but end at D. We see that there are in-
deedn! paths through the decision tree—and so of course it is impossible
to actually draw a graph when the problem is big. But the conceptualiza- tion is often useful to keep in mind. In an optimization situation the task
A B C D B A C D C A B D D A B C C D B D B C B C A C A B D C D B C B C B C A B A
Figure 2.13 Simple permutations of four elements depicted as a tree. The dotted line indicates where branches were omitted due to lack of space.
evaluation or result, whether this is the shortest path from a start to a fin- ish, or any feasible path from a premise to a conclusion, or some other kind of problem.
There are two general ways to search through such a tree data struc- ture.Breadth-first searchinvolves going into the tree one layer at a time, through all branches, and evaluating the result of each partial search. In our four-element example, if we are searching for the shortest path from start to end, we might ask, for instance, starting on A, what is the dis- tance to B, to C, to D? If we started on B, what is the distance to A, to C, to D? and so on. It may be that we can eliminate some routes at this point, just because a step is so large that we are sure not to have the shortest route.Depth-firstsearch comprises going from the start all the way to the end of a path, evaluating one entire proposed solution. Once the end is reached, depth-first search goes back to the first decision node and then searches to the end of the next path, and so on.
Heuristicsare shortcuts in the search strategy that reduce the size of
the space that needs to be examined. We proposed a simple heuristic for breadth-first search in the previous paragraph: abandoning a search as soon as there is reason to expect failure. If we were using a depth-first strategy, we might establish a rule where we abandon a path when its dis- tance becomes longer than the shortest we have found so far. Maybe af- ter a few steps into the tree we will be able to eliminate a proposed path or subtree by this method. There are many clever general heuristics and very many ingenious heuristics that apply only to specific problems.
While “order matters” in permutations, the termcombinationrefers to the situation where some number of elements is selected from a uni- verse of possibilities, without regard to their order. Thus, a universe of {A, B, C, D} contains these combinations: {A}, {B}, {C}, {D}, {A, B}, {A, C}, {A, D}, {B, C}, {B, D}, {C, D}, {A, B, C}, {A, B, D}, {A, C, D}, {B, C, D}, {A, B, C, D}. Since order doesn’t matter in combinations, the combination {A, B, C} is identical to {B, C, A}, {C, A, B}, and so on.
In many situations the problem is defined in such a way that some permutations or combinations are inadmissible; for instance, nodes are connected only to certain other nodes in the graph (there is no road con- necting city B with city F). In these cases the size of the search space is of course reduced, though the problem may be more difficult than a simple permutation problem if at each step the appropriateness or correctness of a branch in the tree needs to be validated.
We can muse upon combinatorial aspects of the great stochastic sys- tems. Even though it is a simple-sounding problem of ordering genetic proteins, natural evolution could not have gone from bacterium directly
to human (which is not to suggest that we are the ultimate achievement of evolution—we just make a good reference point for the present). The search of the space of possibilities that led down to human intelligence needed to discover RNA, DNA, nucleated cell structures, multicellularity, breathing out of water, spines and centralized nervous systems, and so forth in order to reach the social, linguistic primate calledHomo sapiens. At the same time the environment had to change and evolve, affecting and being affected by the new forms that inhabited it. It may have been “technically” possible to jump from bacterium to human in one huge mutational leap, but the probability of that happening was miniscule. Evolution can be seen as a kind of search of the space of possible life- forms, where various adaptations are tried, continuing into the future. An important aspect of this search is that it is conducted in parallel; that is, a number of mutations and genetic combinations are proposed and tested simultaneously throughout the populations of species. Since traits cannot be passed from one species to another, evolutionary search at the macro level can be conceptualized as a branching tree.
Minds as well search their space of ideas in parallel, but the search is not well represented as a tree since the nodes at the ends of the branches are connected to one another. Various cultures explore their own regions of a cognitive parameter space, and within those cultures particular indi- viduals search particular regions of the vast search space of beliefs, atti- tudes, and behaviors. Here and there a culture hits a dead end, and cul- tures interact, passing heuristical information back and forth to one another, and out on the little branches individuals convey information to one another that shapes their searches.
A human born into this world alone, to make sense of what William James called the “buzzing, blooming confusion” of the world, would not be able in a lifetime to understand very much of it at all—if indeed the word “understand” even has meaning for an asocial hominid. Informa- tion sharing is our main heuristic for searching the large space of possible explanations for the world.