Dynamic programming is a technique used to efficiently solve problems involving constrained multi-stage decision processes. Such problems may have a variety of different formulations, but, in general, involve a stage-additive formulation such as (3.1) (Bertsekas,1995). zk(s) =min x,y fN(xN−1,yN−1,sN) + N−1
∑
i=k fi(xi−1,yi−1,si) s. t.gj(xj−1,yj−1,sj−1)≤0 , j=k, . . . ,N hj(xj−1,yj−1,sj−1) =0 , j=k, . . . ,N (3.1)Problem (3.1) comprises N−k decision steps, each involving continuous, xi ∈ Rnc, and discrete, yi ∈ {0, 1}nd, decision variables that influence the state si ∈ Rns.
The formulation is subject to a set of inequality constraints,gj(xj,yj,sj)≤0, equality
constraints,hj(xj,yj,sj) =0, and a stage cost, fi(xi,yi,si).
The sequential structure of (3.1) may be explored by applying the optimality principle, proposed by Bellman (Bellman,1957). In general terms, this principle states that, given an optimal path,Vi, fromi to N, the optimal path fromkto N, that passes byi, also
contains the pathVi. When applied to (3.1), the optimality principle results in recursion
(3.2).
zi(si−1) =xmin
i−1,yi−1 fi(xi−1,yi−1,si) +zi+1(si) s. t.gi(xi−1,yi−1,si−1)≤0
hi(xi−1,yi−1,si−1) =0
(3.2)
Note that problem (3.2) is a sub-problem of (3.1), with decision variables, and constraints, pertaining only to stage i. Bellman (Bellman, 1957) demonstrated that, by recursively solving (3.2) for i = N, . . . ,k, the globally optimal solution of (3.1) is obtained.
The problem addressed in this chapter is a special case of (3.1) and (3.2) where the objective function and constraints are linear, as given by (3.3).
zi(si−1) =xmin
i−1,yi−1Cxi−1+Dyi−1+zi+1(si) s. t.Axi−1+Eyi−1≤b
Lxi−1+Kyi−1=c
yi−1∈ {0, 1}nd
(3.3)
The matrices A,C,D,E,L,K, and vectorsb,cin (3.3) are linear coefficients of appro- priate dimensions.
3.1. Constrained dynamic programming and multi-parametric programming 47 Note that although yi in (3.3) is a binary vector, the formulation may be used to describe an integer vector, yi ∈ (I ∈ Znd), by introducing nd binary variables,
di∈ {0, 1}nd, such thatyi= nd ∑ j=1Ijdi,j, and nd ∑ j=1di,j=1.
The method for constrained dynamic programming by multi-parametric program- ming proposed by Fa´ısca et al. (Fa´ısca et al.,2008), for multi-parametric linear/quadratic problems, involves recursion (3.4), formulated for stageiof a process withNstages.
zi(θi) =minx i−1 fi(xi−1,θi) +zi+1(θi+1) s. t.gi(xi−1,θi)≤0 hi(xi−1,θi) =0 θi = h si−1,xi, . . . ,xN−1 i (3.4)
By considering a parameter vector,θi, that includes both the state vector,si−1, and
the future decisions, xi, . . . ,xN−1, a convex objective function is obtained, and problem
(3.4) may be solved without the need for global optimisation techniques.
Following the same methodology for a mixed-integer linear problem, such as (3.3), a series of multi-parametric mixed-integer linear problems is obtained, of the general form (3.5). zi(θi) =xmin i−1,yi−1Cxi−1+Dyi−1 s. t.Axi−1+Eyi−1≤b+Fθi Lxi−1+Kyi−1=c+Qθi θi ∈Θi,yi−1∈ {0, 1}nd (3.5)
Note that the vector of parameters θi in (3.4) is augmented in (3.5) to include the
discrete optimisation variables,yi+1, . . . ,yN.
To obtain the solution of problem (3.5), several solving algorithms exist in the litera- ture (Acevedo and Pistikopoulos,1997; Pertsinidis et al.,1998; Dua and Pistikopoulos, 2000; Li and Ierapetritou,2007a; Wittmann-Hohlbein and Pistikopoulos,2012a).
The algorithm proposed by Dua et al. (Dua and Pistikopoulos, 2000) is based on the decomposition of problem (3.5) into a mp-lp subproblem, and an associated deterministic milpproblem.
The mp-lp subproblem associated with (3.5) is obtained by fixingyi =yi¯, where ¯yi
is a feasible solution of (3.5). The algorithm for solving the mp-lpsubproblem involves analysing the optimality conditions in the neighbourhood of an optimal solution, for a perturbation in the parameter vector, as outlined in§2.1(Theorem1and Corollary1).
The optimal value of the mp-lpsubproblem,z∗
i(θi), provides an upper bound to the
overall solution of (3.5). To obtain a lower bound, the deterministicmilpproblem (3.6) is solved for each region, CRi, of the map of critical regions where the solution of (3.5) is
zi(θi) = min xi,yi,θiCxi+Dyi s. t.Axi+Eyi ≤b+Fθi Lxi+Kyi=c+Qθi ()
∑
j∈Ji i yji−∑
j∈Lk i yji ≤ |Jik| −1, k=1, . . . ,Ki (4) Cxi+Dyi≤z∗i(θi) θi ∈CRi (3.6)The Ki integer combinations already explored in CRi are divided into sets Jik =
j|yij,k=1, andLk i =
j|yji,k=0, and excluded from the solution by introducing the integer cuts(). The inequality(4)guarantees that the optimal value of (3.6) is not
higher than the current upper bound,z∗ i(θi).
If (3.6) is infeasible, z∗
i(θi) becomes the solution of (3.5) for θi ∈ CRi; otherwise,
problem (3.5) is re-solved, with ¯yset to the newly found integer solution. The resulting optimal value is then compared toz∗
i(θ), using the method presented by Acevedo et
al. (Acevedo and Pistikopoulos,1997), to obtain an update of the current upper bound. The algorithm continues to iterate between themp-lp andmilpsubproblems, until all milpsubproblems are infeasible.
The final solution of (3.5), corresponds to the optimal piece-wise affine functions,
xi(θi)andyi(θi), as well as the map of critical regions for which these relations are valid.
It is possible for the number of critical regions obtained to be significantly large, thus making it computationally expensive to retrieve the corresponding optimal solutions. However, methods have been developed (Tøndel et al.,2003a) to efficiently compute the optimal solutions, by generating a binary search tree.
Note that instead of re-formulating the dynamic programming recursion, (3.3), the original dynamic programming problem (3.1) may be formulated and solved as a single multi-parametric mixed-integer problem, such as (3.5), of larger dimensionality. However, one of the main limitations of multi-parametric mixed-integer programming
remains the exponential complexity of the algorithm (Dua et al., 2002), observed for high-dimensional problems. By re-reformulating the dynamic programming recursion (3.3), a series of lower dimensional multi-parametric mixed-integer linear sub-problems is obtained, that may be efficiently solved. The example in§3.3presents a qualitative complexity analysis that highlights the corresponding improvement in performance.