Classical planning is based on the assumption of complete knowledge of the ini- tial conditions and the effects of actions. Significant advances have been made in recent years in the size and complexity of deterministic problems that can be solved with these planners. However, in real planning problems (Currie and Tate, 1991; Kingston et al., 1996; Drabble et al., 1997; Bernard et al., 1998; Tate et al., 2000; Khan et al., 2003; Meuleau et al., 2009; Fox et al., 2011; Casta ˜no et al., 2014; Bresina, 2015), unexpected events may occur, actions may have unexpected out- comes and the state of the world may not be known with certainty. If a determin- istic planner is used to solve such problems, the execution of the plan may fail because the plan does not take into account the possible contingencies. In order to address this problem, it is important to develop planners capable of handling uncertainty in the domain (Bonet and Geffner, 2005; Little and Thi´ebaux, 2006; Yoon et al., 2007, 2008, 2010; Keyder and Geffner, 2008b; Kalyaman and Givan, 2008; Buffet and Aberdeen, 2009; Teichteil-K ¨onigsbuch et al., 2010; Kolobov et al., 2012; Keller and Helmert, 2013).
A line of research dealing with planning problems under uncertainty is prob- abilistic planning, which describes the uncertainty using probability distribu- tions. Probabilistic planning techniques have been developed to address prob- lems where the uncertainty is fully observable, partial observable, or not observable. Probabilistic PDDL (PPDDL) (Younes et al., 2005) is an extension of the PDDL language capable of representing probabilistic planning problems by allowing probabilities on initial conditions and on action outcomes. The semantic of a PPDDL planning problem is given in terms of a discrete-time Markov Decision Processes (Puterman, 1994). To define probabilistic planning problems, PPDDL describes probabilistic actions as in PDDL with the addition of probabilistic out- comes on action definitions:
where each ei is a PDDL effect that occurs with probability pi. A probabilistic effect represents a distribution over deterministic effects wherePk
i=1pi = 1. A description in PPDDL of a simple probabilistic Logistics domain and problem is shown in Figure 2.9. The source of uncertainty in this domain is in action drive: (:action drive
:parameters (?t - truck ?from - location ?to - location ?pkg - package) :precondition (and (connected ?from ?to) (at ?from ?t) (in ?pkg ?t)
(verified ?pkg ?t) (not (flattire)))
:effect (and (not (at ?from ?t)) (at ?to ?t) (probabilistic 0.4 (flattire)))) where action effects are (not (at ?from ?t)), (at ?to ?t), and (flattire), but the latest only happens with a probability of 0.4. This can be seen as two different action outcomes:
• o1, where (not (at ?from ?t)) and (at ?to ?t) happens with probability 0.6. • o2, where (not (at ?from ?t)), (at ?to ?t), and (flattire) happens with probabil-
ity 0.4.
There is a second source of uncertainty in action verify: (:action verify
:parameters (?pkg - package ?t - truck ?l - location ?to - location) :precondition (and (at ?l ?t) (in ?pkg ?t)
:effect (and (probabilistic 0.8 (verified ?pkg ?t))))
where the action’s effect is (verified ?pkg ?t), but it only happens with a probabil- ity of 0.8. Otherwise, the action does nothing.
Figure 2.10 shows a simple probabilistic Logistics problem where there is a package and a truck at location a, and the package needs to be delivered to lo- cation c. The package can be loaded on the truck. The truck can drive between locations when it does not have a flat tire and it is verified that the package is inside the truck. The verification of the package may fail with probability 0.2. In addition, a tire may go flat during the drive with a probability of 0.4. A flat tire can be changed, if the truck is at a locations a or d where there is a spare tire. Finally, before unloading the package from the truck, it must be scanned.
Probabilistic planning is an active research field. Several different techniques are currently being used to solve probabilistic planning problems. We classify these techniques in four main groups that are described below.
Solving Markov Decision Processes
Markov Decision Processes (MDPs) (Puterman, 1994) is a popular family of mod- els for analyzing probabilistic planning problems. An MDP is defined as a tuple M = h S, A, T, C, G iwhere S is a finite set of all possible states; A is a finite set of all actions an agent can take; T : S × A 7→ S0is a transition probability function
(define (domain logistics)
(:requirements :strips :typing :probabilistic-effects) (:types truck package location)
(:predicates (connected ?a1 ?a2 - location) (at ?l - location ?pkg - package) (at ?l - location ?t - truck) (in ?pkg - package ?t - truck) (scanned ?pkg - package ?t - truck) (verified ?pkg - package ?t - truck) (flattire) (spare ?l - location) (hasspare))
(:action drive
:parameters (?t - truck ?from - location ?to - location ?pkg - package)
:precondition (and (connected ?from ?to) (at ?from ?t) (in ?pkg ?t) (verified ?pkg ?t) (not (flattire))) :effect (and (not (at ?from ?t)) (at ?to ?t) (probabilistic 0.4 (flattire))))
(:action get-tire
:parameters (?l - location ?t - truck) :precondition (and (at ?l ?t) (spare ?l)) :effect (and (not (spare ?l)) (hasspare)))
(:action change :precondition (hasspare)
:effect (and (not (hasspare)) (not (flattire))))
(:action load
:parameters (?pkg - package ?l - location ?t - truck) :precondition (and (at ?l ?t) (at ?l ?pkg))
:effect (and (in ?pkg ?t) (not (at ?l ?pkg))))
(:action verify
:parameters (?pkg - package ?t - truck ?l - location) :precondition (and (in ?pkg ?t) (at ?l ?t))
:effect (and (probability 0.8 (verified ?pkg ?t))))
(:action scan
:parameters (?pkg - package ?t - truck)
:precondition (and (in ?pkg ?t) (verified ?pkg ?t)) :effect (and (scanned ?pkg ?t)))
(:action unload
:parameters (?pkg - package ?t - truck ?l - location) :precondition (and (at ?l ?t) (scanned ?pkg ?t)) :effect (and (not (in ?pkg ?t)) (at ?l ?pkg)))
(define (problem logistics-p01) (:domain logistics)
(:objects a b c d e - location trk - truck pkg - package (:init (connected a b) (connected a d) (connected b c)
(connected d e) (connected e c) (at a trk) (at a pkg) (spare a) (spare d) (spare e) (not (flattire))) (:goal (at c pkg)))
Init Goal a b c d spare pkg trk spare a b c d pkg
Figure 2.10: Initial and goal states for a simple probabilistic Logistics problem.
that returns a new state S0 according to a probability distribution PT(S0|S, A); C : S × A 7→ R is a cost function that gives a finite numeric cost value C(S, A), which represents the cost for taking action A in S; and G is the goal state. A solu- tion to an MDP is a policy π : S 7→ A, which is a mapping from states to actions. (In other words, a rule for action selection.) The goal is to find a policy that min- imizes the expected cost. An optimal policy π∗ can be found using a Dynaminc Programming (DP) algorithm such as Value Iteration (Bellman, 1957) or Policy Iteration (Howard, 1960).
Value Iteration is a DP method that computes an optimal policy and its value starting with an initial evaluation function V0and a parameter ε for detecting con- vergence to an optimal evaluation function. For each state s ∈ S, the algorithm first improves f by performing dynamic programming update as follows:
Vn(s) = min a ∈ A(s) ( cs(a) + X s0∈ S PT(s0|s, a) Vn−1(s0) ) (2.13)
where cs(a) is the expected cost of taking action a in state s, PT(s0|s, a) is the probability that taking action a in state s results in state s0, and Vn−1(s0) is the value of V (s0)at iteration n − 1.
This process is repeated until a convergence is reaches. In other words, when Vnis less than or equal to ε. Then, a policy can be extracted from f as follows:
π(s) = argmin a ∈ A(s) ( cs(a) + X s0∈ S PT(s0|s, a) Vn−1(s0) ) (2.14)
Policy Iteration is another DP method that computes an optimal policy start- ing with an initial policy π. It interleaves policy improvement with policy evalu- ation. It first computes the evaluation function Vπn−1 for policy π by solving the following linear equation:
Vnπ(s) = cs(π(s)) + X s0∈ S
Then, for each state i ∈ S it improves the policy as follows: π0(s) = argmin a ∈ A(s) ( cs(a) + X s0∈ S PT(s0|s, a) Vn−1π (j) ) (2.16)
This process is repeated until convergence is reached. In order words, when π0is the same as π.
The disadvantage of Value Iteration and Policy Iteration is that they evaluate the entire state space. For this reason, Barto, Bradke, and Singh (1995) proposed the real-time dynamic algorithm (RTDP) that minimizes the search space of DP. The RTDP algorithm involves a sequence of trials to investigate choices of actions for each state. Each trial consists of (1) selecting the best action a that minimizes the cost of reaching the goal starting in s, (2) updating the value Vn(s)using Equa- tion 2.13, and (3) transitioning to a successor state s0 under a. A significant draw- back to RTDP is the lack of convergence detection. To overcome this problem, Bonet and Geffner (2003a) developed Labeled RTDP (LRTDP), which works in the same way as RTDP, but guarantees convergence by labeling visited states.
One feature that almost all approaches share is that all potential action out- comes in the plan are taken into account. That is, the plan includes a contingency branch for every possible outcome that may occur, given the actions in the plan. These approaches generate robust plans, but can be computationally expensive.
Using contingency planning
These approaches use planning graphs to compute estimates of probability that propositions can be achieved and actions can be performed. This information can be used to guide a probabilistic planner toward the most likely plan for achiev- ing the goals. In particular, Blum and Langford (1999) extend the Graphplan representation to use it for probabilistic planning, and developed PGraphplan and TGraphplan. PGraphplan is a planner based on a forward-chaining search through a plan graph. It finds optimal (non-concurrent) contingency plans via dy- namic programming using the information stored in the plan graph to prune the search. Since mutual exclusion relations computed in a plan graph are not help- ful in a forward search, PGraphplan propagates information backwards through a plan graph to identify states from which the goal is probably unreachable. TGraphplan is a minor extension of the original Graphplan algorithm. It finds a non-optimal single path to the goal with highest probability. During the TGraph- plan backwards search, the probability of the path is determined by recursively multiplying the probability of a step succeeding by the probability of the partial
plan already explored. Paragraph (Little and Thi´ebaux, 2006) is another plan- ner that also applies probabilistic plan graph, nogood learning, mutex reasoning, and goal regression techniques to probabilistic planning. It generates an optimal contingent plan by concatenating sub-trajectories generated by the plan graph. Bryce, Kambhampati, and Smith (2006) also use a plan graph to compute es- timates of probability. However, they use an AO* algorithm in seeking a plan solution.
Translation into deterministic planning problems
These approaches translate the given planning problem into a classical planning problem that is then solved by a classical planner (Palacios and Geffner, 2009). Other approaches combine the translation-based technique with replanning (Al- bore et al., 2009), or replanning and sampling (Brafman and Shani, 2012), getting better results. It is important to note that these planners deal with uncertainty in the initial state and conditional effects. Also, for the most part these planners only deal with disjunctive uncertainty not probabilistic uncertainty.
Determinization-based techniques
Determinization consists of transforming the given probabilistic planning prob- lem into a deterministic planning problem and using heuristic functions based on relaxed plans to guide a deterministic planner in the search for a determin- istic plan. As a result, these planners generate a solution that is executed un- til the observed state differs from what is expected according to the next step in the solution. If this happens, replanning is performed to seek another plan solution. The insight behind this idea is that a deterministic planning problem can be solved more efficiently than the equivalent probabilistic MDP. The FF- Replan planner (Yoon et al., 2007) has been the pioneer in this paradigm, and it has proven very successful on many problems. It proposes two different methods of determinization:
• single-outcome determinization, which generates a deterministic action for each probabilistic action by selecting the outcome with the highest probability. To illustrate this, consider the probabilistic action (drive) in Figure 2.9. The most likely outcome is that the car successfully drive from one position to another. This determinization technique generates action (drive-1) for that outcome:
(:action drive-1
:parameters (?t - truck ?from - location ?to - location ?pkg - package) :precondition (and (connected ?from ?to) (at ?from ?t) (in ?pkg ?t)
(verified ?pkg ?t) (not (flattire))) :effect (and (not (at ?from ?t)) (at ?to ?t)))
• all-outcomes determinization, which generates a deterministic action for each outcome of a probabilistic action. To illustrate this, consider again the prob- abilistic action (drive) in Figure 2.9. The probabilistic actions is determinized generating action (drive-1), where the car successfully drives from one loca- tion to another, and action (drive-2), where the car successfully drives from one location to another, but gets a flat tire:
(:action drive-1
:parameters (?t - truck ?from - location ?to - location ?pkg - package) :precondition (and (connected ?from ?to) (at ?from ?t) (in ?pkg ?t)
(verified ?pkg ?t) (not (flattire))) :effect (and (not (at ?from ?t)) (at ?to ?t)))
(:action drive-2
:parameters (?t - truck ?from - location ?to - location ?pkg - package) :precondition (and (connected ?from ?to) (at ?from ?t) (in ?pkg ?t)
(verified ?pkg ?t) (not (flattire))) :effect (and (not (at ?from ?t)) (at ?to ?t) (flattire)))
The first method may neglect important possible effects, while the second method considers every possible probabilistic outcome and may produce plans that rely on low probability outcomes.
Despite the success of FF-Replan, it does not make use of the probabilistic in- formation in the domain description, which may result in frequent replanning. For this reason, most recent probabilistic planners follow the same line as FF- Replan, but deal with the probabilities of action outcomes (Kalyaman and Givan, 2008; Keyder and Geffner, 2008b; Wu et al., 2011). In particular, the work done by Jim´enez, Coles, and Smith (2006) turns the probability information into costs. They make use of all-outcomes-determinization technique and for each new de- terminized action created, they transform the probability of the outcome into a cost equal to the negative logarithm of the probability. Then, they search for an optimal plan using a numeric deterministic planner that minimizes cost.
These approaches perform fast planning. However, they cannot anticipate possible deviations from the linear plan that is generated by the deterministic planner and, therefore, get stuck in dead-end states. FFH+(Yoon et al., 2008, 2010) is an improved version of FF-Replan that avoids that problem. It applies Hind- sight Optimization, and on-line anticipatory strategy, for action selection. Deter- minization in hindsight samples actions according to their distribution (incorpo- rates probability information). Then, it solves each determinized problem using an off-the-self deterministic planner at each step, and analyzes their solutions to select the action that leads to the best outcome. RFF (Teichteil-K ¨onigsbuch et al., 2010) is another determinized-based probabilistic planner that attempts to avoid dead-ends states in advance as FFH+. RFF performs all-outcomes-determinization and then uses this relaxation to compute an off-line policy by generating con- secutive execution paths leading to the goal from the initial states using the FF planner. The resulting policy has low probability of failing.