Planning as heuristics search

A classical planner aims to automatically solve a deterministic planning prob- lem by exploring the state-space of the problem to find a path from I to G. One optimal way to find an optimal solution to a planning problem is to perform an

exhaustive search over the state-space and get the best (optimal) solution. How- ever, this is impractical for a large size states-spaces. As mentioned previously, reachability analysis may help to reduce the state-space, although it is insuffi- cient. A more effective way to solve deterministic planning problems is to use heuristic search algorithms (Hart et al., 1968; Nilsson, 1980; Hoffmann and Nebel, 2001; Hansen and Zilberstein, 2001), which have been used in most recent clas- sical planners (Bonet and Geffner, 2001; Hoffmann and Nebel, 2001; Hoffmann, 2003; Gerevini et al., 2003; Do and Kambhampati, 2003; Chen et al., 2006; Keyder and Geffner, 2007, 2008a; Keyder et al., 2010; Richter and Westphal, 2010; Keyder et al., 2012).

A heuristic search algorithm is a technique that uses a heuristic to find the sequence of actions that reach the goal from the initial state. The search space consists of a set of nodes, which represent a description of a state s in form of propositions. There is a unique node s0 known as the initial state node. Nodes are connected by arcs, which represent an action a that was used to reach a state s0 from state s. s0 is known as the successor node of s. A basic heuristic search procedure explores the search space by ranking all the current reachable nodes according to the following heuristic function:

f (n) = cost(s) + h(c(s, g)) (2.1)

where cost(s) is the cost of the explored part, and h(c(s, g)) is the estimated cost of reaching a goal state g from a given state s (Pearl, 1984). Therefore, f (n) is used to guide the search towards the most promising nodes. The algorithm starts from s0 by expanding and computing f (n) for each successor of s0. These successors are then ranked according to their f (n) in an open list. At each step, the most promising successor is removed from the open list to be inserted in a closed list, while its successors are added to the open list, ordered by the value of f (n). In order to avoid repeated states, it is verified if the current successor is present in the open list or closed list, which means that it has already been generated. If it is present in the closed list or in the open list with a worse f (n), the value and the action associated with the node are updated, and if it was present in the closed list, the node is moved to the open list.

Pearl (1984) defines a heuristic as a method to decide among alternative courses of action in order to find the most effective way to achieve a goal. A heuristic needs to be simple to compute and effective to discriminate among the different alternatives. Although it does not always guarantee optimality, it has been an efficient method in combination with heuristic search algorithms to explore the state-space even when it is extremely large.

In heuristic planning techniques, the search space is assumed to be an OR graph where a path from an initial state to a goal state needs to be found. A graph consists of a set of nodes, which in the planning context represent the description of a state s in the form of propositions. Nodes are connected by arcs, which in the planning context represent an action a that was used to reach a state s0 from state s. In the search of this path using heuristic planning techniques, a heuristic takes the form of heuristic estimator (h), which is the estimated cost of reaching a goal state g from a given state s through the course of action beginning with a, i.e., h(s) = c(s, a). The perfect heuristic estimator, h∗, provides the optimal (or cheapest) cost from a state s to a goal state g. That is, h∗(s) = min{c(s, a)}. There is always an optimal path to reach g from s. Essentially, in order to guarantee an optimal solution, h should be optimistic and never overestimate the cost of reach- ing the goal. Such heuristics are called admissible heuristics. A heuristic estimator his admissible when:

h(s) ≤ h∗(s) ∀ s (2.2)

Admissible heuristic estimators in combination with heuristic search algo- rithms produce optimal solutions, but may take a long time for large state-spaces. A drawback of an admissible heuristic estimator is that it may spend a significant amount of time discriminating among the different courses of action for some problems. In those cases where admissibility is harmful to the search process, non-admissible heuristic estimators can be used, which may lead the problem towards suboptimal solutions. Non-admissible heuristic estimators can overesti- mate the cost to reach the goal, which means that potential courses of action that actually have a lower cost may not be explored. For this reason, non-admissible heuristics can produce suboptimal solutions, but make often the search more tractable.

Heuristic functions are commonly derived from simplified models of the orig- inal problem, which are generated by removing constraints and by making as- sumptions (Pearl, 1984). This is known as relaxation. A common relaxation-based heuristic in planning is the delete relaxation, which consists of ignoring delete ef- fects. In other words, in an original problem, any applicable action a in a state s produces a new state s0 where s0 = {s − (add(a) ∪ del(a))}. If delete effects are ignored, s0 = {s + add(a)}. This means that a proposition holds after the first level at which it can be achieved. Relaxation-based heuristics are admissible if the cost of the relaxation is lower than the optimal cost. The following heuristic functions are based on this relaxation idea.

The max heuristic

The max heuristic hmax(Bonet and Geffner, 2001) computes an estimation of cost, which is defined by the following recursive equation:

hmax(p, s) def=          0 if p ∈ s argmin a ∈ O(p)  cost(a) + max q ∈ prec(a)h max_{(q, s)}  otherwise (2.3)

where hmax(p, s) refers to an estimate of the cost of achieving a proposition p from state s. This estimate is the minimum cost among all the actions a that add p, which, in turn, is defined as the maximum cost among all preconditions of a. The max heuristic is admissible since the cost of achieving a set of propositions cannot be lower than the cost of achieving any proposition of the set.

The additive heuristic

The additive heuristic h+(Bonet and Geffner, 2001) computes a cost approxima- tion, which is defined by the following recursive equation:

h+(p, s) def=          0 if p ∈ s argmin a ∈ O(p)  cost(a) + P q ∈ prec(a) h+_{(q, s)}  otherwise (2.4)

where h+(p, s)refers to an estimate of the cost of achieving a proposition p from state s. This estimate is the minimum cost among all the actions a that add p, which, in turn, is defined as the sum of the cost of the individual preconditions of a. This estimation of cost assumes that subgoals are independent. When this as- sumption is true, that is, when preconditions of an action do not interfere with each other, the additive heuristic is admissible in the delete relaxation. On the other hand, when this assumption is not true, that is, when preconditions of an ac- tion are synergistic with each other, it may overestimate the cost. For this reason, the additive heuristic is not admissible. Therefore, the estimated cost computed is an approximation of the optimal cost.

The relaxed plan heuristic

The relaxed plan heuristic hF F (Hoffmann and Nebel, 2001) computes a cost ap- proximation by computing a plan, which is not guaranteed to be optimal. This is done using a construction plan graph procedure like is described in Section 2.1.2. Since delete relaxation is assumed, a relaxed plan graph with no mutex is built. The relaxed plan graph is used to extract a plan π starting from state s, where the length of π gives an approximation of the estimated cost of achieving the goal from s.

FunctionRELAXEDPLANEXTRACTION(Gl)

l ≡ number of levels in the relaxed plan graph Gl ≡ the set of goals at level l in the relaxed plan graph g ≡ a goal proposition

S ≡ the set of propositions of Glalready achieved Al ≡ the set of actions at level l for an specific goal a ≡ an action

la ≡ the level in which action a is first reached π ≡ a relaxed plan

Lπ ≡ the length in number of actions of π

1. π ← ∅ 2.while l > 0 do 3. S ← ∅ 4. Gl−1← ∅ 5. while Gl6= ∅ do 6. g ← g ∈ Gl 7. if g /∈ S then do 8. Al← {a : g ∈ add(a) ∧ la< l } 9. a ← argmin a∈Al { | prec(a) | } 10. S ← S ∪ add(a) 11. Gl−1← Gl−1∪ prec(a) 12. π ← π ∪ {a} 13. Gl← Gl− {g} 14. l ← l − 1 15.return Lπ

Figure 2.5: The relaxed plan extraction pseudo-algorithm.

Figure 2.5 shows the high-level algorithm used to extract a relaxed plan. The algorithm starts with the propositions corresponding to the goals at level l. For each goal proposition g, the algorithm selects an action from level l − 1, that sup- ports g. If there is more than one supporter, the process selects the one with the lowest number of preconditions. Each time an action is selected, all its positive effects are marked as achieved at level l, to prevent double check goals from level

l, and its preconditions are added as goals for the l − 1 level. The algorithm re- peats the same process on the l − 1 level of the plan graph, with preconditions of the actions selected at actions’ layer l − 1 as the goals for the l − 1 level. The search succeeds when level 0 is reached, that is, the initial state. The sequence of selected actions forms relaxed plan. The heuristic value is the length, or number of actions, of the relaxed plan.

To illustrate this algorithm, considerer the simple Logistics problem described in Section 2.1. Figure 2.6 shows the plan graph for the problem. The goal set at level 4 contains the single proposition (at b pkg) whose supporter is action (unload pkg trk b) at level 3. Propositions (scanned pkg trk) and (at b trk), which are the preconditions of (unload pkg trk b), are the goals at level 3. The action (scan pkg trk) supports the goal (scanned pkg trk), and adds (in trk pkg) and (verified pkg trk) as goals at level 2. The action (drive trk a b) supports the goal (at b trk), and adds (at a trk), as well as (in trk pkg) and (verified pkg trk) that were already added by (scan pkg trk). The goal (at a trk) and (in trk pkg) at level 2 have the (noop) action as supported, while (verified pkg trk) has action (verify pkg a trk), which adds propositions (at a trk) and (in trk pkg) as goals at level 1. (These two propositions were already added by the (noop) action.) Again, (at a trk) is supported by the (noop). The proposition (in trk pkg) is supported by action (load pkg a trk), which adds propositions (at a trk) and (at a pkg) goals at level 0. Finally, propositions (at a trk) and (at a pkg) are true at level 0, which means there is a valid solution. The relaxed plan consists of actions (load pkg a trk), (verify pkg a trk), (scan pkg trk), (drive trk a b), and (unload pkg trk b). Therefore, for this example the heuristic value is 5.

P0 at a trk at a pkg A0 load pkg a trk P1 at a trk at a pkg in trk pkg A1 load pkg a trk verify pkg a trk P2 at a trk at a pkg in trk pkg verified pkg trk A2 load pkg a trk verify pkg a trk scan pkg trk drive trk a b P3 at a trk at a pkg in trk pkg verified pkg trk scanned pkg trk at b trk A3 load pkg a trk verify pkg a trk scan pkg trk drive trk a b unload pkg trk b P4 at a trk at a pkg in trk pkg verified pkg trk scanned pkg trk at b trk at b pkg

Unlike the additive heuristic and the max heuristic, the original relaxed plan heuristic does not take cost information into account, and considers unit action costs. Nevertheless, it is possible to easily extend this to action costs by summing the cost of the actions in the relaxed plan.

The higher-order heuristic

The higher-order heuristic hm _{(Haslum and Geffner, 2000) computes an admis-} sible cost approximation for regression search. It is a generalization of the max heuristic that computes estimates of cost for sets of m propositions from the initial state s0. It is defined by the following equation:

hm(A)def=                    0 if A ∈ s0 min

hB,ai ∈ R(A)cost(a) + h

m_{(B) if |A| ≤ m and A * s} 0 max B ⊂ A, |B| = mh m_{(B) if |A| > m} (2.5)

where hm_{(A)refers to the cost of achieving a state s where a set of propositions A} of size m is true, s0refers to the initial state definition, and R(A) refers to the set of pairs hB, ai where B is the resulting state of regressing A through a, defined as R(B, a) = {A − add(a)} ∪ prec(a).

The hm heuristic is admissible because hm(B) ≤ h∗(B)for every B, and it is true for size m of the set. However, it presents a trade-off between accuracy and efficiency. Higher values of m are more accurate, but more expensive to compute. The computation of hm is a low-order polynomial in Nmwhere N is the number of propositions in the problem, and m the size of the considered sets. For practical purposes, it is limited to m = 2 or m = 3. When the size of m is one, the heuristic hm_{is equivalent to the max heuristic. When the size of m is two, the heuristic h}m computes the estimated cost between two propositions p and q as follows:

h2({p, q}) = min                      min a ∈ O(p∧q) cost(a) + h 2_(prec(a)); min a ∈ O(p|q) cost(a) + h 2_{(prec(a) ∪ {q});} min a ∈ O(q|p) cost(a) + h 2_{(prec(a) ∪ {p})} (2.6)

where h2({p}) = min

a∈O(p) cost(a) + h

2_{(prec(a)), O(p ∧ q) refers to the set of actions} that achieves p and q simultaneously, O(p|q) refers to the set of actions that adds pand does not add or delete q, and O(q|p) refers to the set of actions that adds q and does not add or delete p.

The set-additive heuristic

The set-additive heuristic hs

a(Keyder and Geffner, 2007) is a modification of the additive heuristic that produces relaxed plans taking the cost information into consideration. It can be defined by the following recursive equation:

hs_a = πa(p, s) def =      {} if p ∈ s πa(ap, s) otherwise (2.7) ap = argmin a ∈ O(p)  Cost(πa(a, s))  πa(a, s) = {a} [  ∪ q ∈ prec(a)π(q, s)  Cost(πa(a, s)) = X a0_∈π(a,s) cost(a0)

where πa(p, s)refers to a relaxed plan for p from s. During the construction of a relaxed plan graph for a state s, the additive heuristic is used to propagate the cost and keep track of the actions that minimize each proposition in the relaxed plan graph. During the construction of a relaxed plan π(s) in s, actions that minimize the cost of each subgoal are selected. The estimated relaxed plan for a set of propositions in s is the union of the relaxed plans for each proposition in s. The cost approximation is the sum of the costs of the actions in π(s). That is:

hs_a(s) = X a ∈ π(s)