ONLINE APPENDIX
for
Operations Systems with Discretionary Task Completion
Wallace J. Hopp, Seyed M.R. Iravani, Gigi Yuen
Department of Industrial Engineering and Management Sciences Northwestern University, Evanston, IL
August 2005
Single-Job Class Two-server MDP Model
The following MDP represents a single-job class system with two-servers (with identical or different speeds). It is used in Section 6.
• State Space includes states (n, t
1, t
2), where n is the number of jobs in the system, and t
qis the number of time intervals that the job under service has been worked on in server q. Both n and t are non-negative integers.
• Decision epochs are the beginning of every period.
• Action space includes Keep (K), Release & Work (RW), Release & Idle (RI) and Idle (I) for each server. The 16 unique combinations of the two workers’ actions constitute the system’s action space. Action “Keep” requires the worker to continue working on the current job for one more period. Action “Release & Work” requires that the worker stop working on the current job, and start a new job immediately. Action “Release & Idle” requires the worker to stop working on the current job and become idle for the next time period. Action “Idle”
requires worker to remains idle for one more period.
The optimality equation for the case where n ≥ 4, t
1≥ t
min+ 1 and t
2≥ t
min+ 1 is:
δτ g + V (n, t
1, t
2) = max
λδτ V (n, t
1+ 1, t
2) + (1 − λδτ )V (n + 1, t
1+ 1, t
2+ 1) : (K-K)
− nδτ
λδτ V (n − 1, t
1+ 1, 1) + (1 − λδτ )V (n, t
1+ 1, 1) : (K-RW)
− (n − 1)δτ + f
2(t
2)
λδτ V (n − 1, t
1+ 1, 0) + (1 − λδτ )V (n, t
1+ 1, 0) : (K-RI)
− (n − 1)δτ + f
2(t
2)
λδτ V (n − 1, 1, t
2+ 1) + (1 − λδτ )V (n, 1, t
2+ 1) : (RW-K)
− (n − 1)δτ + f
1(t
1)
λδτ V (n − 2, 1, 1) + (1 − λδτ )V (n − 1, 1, 1) : (RW-RW)
− (n − 2)δτ + f
1(t
1) + f
2(t
2)
λδτ V (n − 2, 1, 0) + (1 − λδτ )V (n − 1, 1, 0) : (RW-RI)
− (n − 2)δτ + f
1(t
1) + f
2(t
2)
λδτ V (n − 1, 0, t
2+ 1) + (1 − λδτ )V (n, 0, t
2+ 1) : (RI-K)
− (n − 1)δτ + f
1(t
1)
λδτ V (n − 2, 0, 1) + (1 − λδτ )V (n, 0, 1) : (RI-RW)
− (n − 2)δτ + f
1(t
1) + f
2(t
2)
λδτ V (n − 2, 0, 0) + (1 − λδτ )V (n − 1, 0, 0) : (RI-RI)
− (n − 2)δτ + f
1(t
1) + f
2(t
2) where g is the optimal average profit per unit time.
The optimality equations for the cases where n < 4 and t
1, t
2≥ t
minfollow similar format but action Release & Work may not be applicable.
For cases where one or both of the servers is idling (t = 0), an idle server does not have the
options to Release and Work or Release and Idle but does have the option to remain idle. For
example, the optimality equation for the case where n ≥ 2, t
1≥ t
minand t
2= 0 is:
δτ g + V (n, t
1, 0) = max
λδτ V (n, t
1+ 1, 1) + (1 − λδτ )V (n + 1, t
1+ 1, 1) : (K-K)
− nδτ
λδτ V (n, t
1+ 11, 0) + (1 − λδτ )V (n + 1, t
1+ 1, 0) : (K-I)
− nδτ
λδτ V (n − 1, 1, 1) + (1 − λδτ )V (n, 1, 1) : (RW-K)
− (n − 1)δτ + f
1(t
1)
λδτ V (n − 1, 1, 0) + (1 − λδτ )V (n, 1, 0) : (RW-I)
− (n − 1
δτ + f
1(t
1)
λδτ V (n − 1, 0, 1) + (1 − λδτ )V (n, 0, 1) : (RI-K)
− (n − 1)δτ + f
1(t
1)
λδτ V (n − 1, 0, 0) + (1 − λδτ )V (n, 0, 0) : (RI-I)
− (n − 1)δτ + f
1(t
1) where g is the optimal average profit per unit time.
For cases where 1 ≤ t ≤ t
min, optimality equations can also be written in similar fashion by
considering only actions Keep and Idle.
Numeric Summary on the Effects of Capacity via Increasing the Number of Workers
The following summary is the result from the increase in number of workers study in Section 5.2.
Table 3. The effects of capacity via increasing the number of servers
Min. Traffic Holding cost Single Server Two Servers % Change % Change Intensity ratio E[Profit] E[Q] E[Profit] E[Q] in E[Profit] in E[Q]
0.2 0.01 8.7 0.87 9.0 0.78 3% -10%
0.04 6.8 0.50 7.1 0.51 5% 3%
0.4 0.01 15.8 1.54 17.6 1.55 11% 0%
0.04 12.5 0.87 14.0 0.99 12% 14%
0.6 0.01 21.2 1.94 25.7 2.26 22% 16%
0.04 16.9 1.12 20.6 1.39 22% 24%
MDP Model and Input Parameter
for the Stochastic Value-Time Function Analysis
The following MDP represents a single-job class system with stochastic process times. It is used in Section 7.
• State Space includes states (n, t, q), where n is the number of jobs in the system, t is the number of time intervals the job under service has been worked on, and q is the type of value-time curve for the job under service. The variables n and t are non-negative integers.
• Decision epochs are the beginning of each period.
• Action space includes actions Keep (K), Release (R)
The optimality equation of the MDP model for the case where and n ≥ 2 and t ≥ t
min+ 1 is:
δτ g + V (n, t, q) = max
λδτ V (n + 1, t + 1, q) + (1 − λδτ )V (n, t + 1, q) − nhδτ : (K) λδτ £
γV (n, 1, 1) + (1 − γ)V (n, 1, 2) ¤
: (R) + (1 − λδτ ) £
γV (n − 1, 1, 1) + (1 − γ)V (n − 1, 1, 2) ¤
− (n − 1)hδτ + f
q(t)
where g is the optimal average profit per unit time. The optimality equations for the case where n = 1 can be written in similar fashion. For the cases where t ≤ t
min, optimality equations can also be written in similar fashion by considering only action Keep.
Using the same parameters we defined for the single-class study in Section 4.3, we considered the following parameter settings for each value-time function setting q = 1, 2:
• Shape of value function A
q: We considered values A
q= 0.24, 0.54 and 0.79.
• Minimum Traffic Intensity ρ
min: We considered values from 0 to 0.8 in increments of approx- imately 0.2.
• Holding Cost to Maximum Value Ratio h
q/b
q: We considered values 0.01, 0.02, and 0.04.
• Minimum quality f
minq/b
q: We considered the values 20%, 40% and 60%.
• Process flexibility φ
q: We considered values ranging from 0.6 to 1 in increments of approxi- mately 0.2.
Specific to the stochastic system, we considered the following parameters that describe the rela-
tionship between the jobs with different value-time curves:
• Probability of a Type 1 Arrival γ: We set this parameter to γ = 0.25, 0.5 and 0.75.
• Ratio of Profit Function Shape Parameter A
1/A
2: We considered values 0.25, 1 and 4. This
ratio describes the relative shapes of the value-time functions. The greater the ratio, the
faster the type 1 value-time function increases over time (df
1(t)/dt) relative to the type 2
value-time function (df
2(t)/dt).
Linear Programming Model, Input Parameter Ranges and the Computation Limitation
for the Partial Information System
We use LP to obtain the optimal policy under accurate holding costs for the Partial Information case (Section 8.1). To accomplish this, we define state i in the LP by vector (x
1, x
2, x
3, . . . , x
ni, t
i, a
i), where x
j(x
j= 1 or 2) is the job type at the j
thlocation in the queue, t
iis the amount of time spent on the current job, a
i∈ {Keep, Release} is the action chosen in state i, and n
iis the total number of jobs in the system in state i. Note that x
1indicates the type of the task under process.
We define:
• yi as the probability of being in state i at any instant, and Y as the vector of yis,
• Ci as the profit resulting from being in state i for duration δτ , which is the revenue generated from the job at release minus the sum of holding costs of type 1 and type 2 jobs in the system per δτ .
• pij as the transition probability from state i to state j, with P representing the square transition probability matrix composed of pij, and
• zi as a binary variable which has value 1 when the probability of being in state i is greater than zero, and is zero otherwise.
We formulate the LP for obtaining the expected optimal profit in a partial information system as follows:
M ax X
i
C
iy
is. t. X
i
y
i= 1 (2)
Y = YP (3)
y
i≤ z
i∀ i (4)
y
k≥ M z
i∀ i, k, where n
i= n
kand a
i= a
k(5)
y
i≥ 0 for all i (6)
where M is a very small positive number with magnitude in the range from 10
−7to 10
−9. Con-
straints (2), (3), and (6) are standard MDP constraints (see Hiller and Lieberman 2001). Con-
straints (4) and (5) are added to force the system to ignore the composition of the queue in
optimizing the profit. This is done by using a binary variable z
ifor each state i. When y
iis
positive, the corresponding action is optimal and Constraint (4) gives z
ithe value of one. Then,
since M is a very small positive number, Constraint (5) requires all the states with the same queue
length and action to have their y
ito be positive and smaller than one. To satisfy these constraints, all states with the same queue length must use the same action. Despite the fact that information on queue composition is available (to ensure correct holding cost calculation), the system makes decisions as if it is unaware of it and bases its decision only on queue length. As a result, it behaves exactly as if it had only the information available in the partial information case.
Using the same parameters we defined for the single-class study in Section 4.3, we studied the following parameter settings for each job class q = 1, 2:
• Shape of value function A
q: We considered values A
q= 0.24, and 0.79.
• Minimum Traffic Intensity ρ
min: We considered values from 0 to 0.8 in increments of approx- imately 0.2.
• Holding Cost to Maximum Value Ratio h
q/b
q: We considered values 0.01, and 0.04.
• Minimum quality f
minq/b
q: We considered the values 20%, and 60%.
• Process flexibility φ
q: We considered values ranging from 0.6 to 1 in increments of approxi- mately 0.2.
Specific to the multi-class system, we consider the following parameters that describe the relation- ship between the jobs with different processing times:
• Probability of a Type 1 Arrival γ: We set this parameter to γ = 0.25, 0.5 and 0.75.
• Ratio of Profit Function Shape Parameter A
1/A
2: We considered values 0.25, 1 and 4. This ratio describes the relative shapes of the value-time functions. The greater the ratio, the faster the type 1 value-time function increases over time (df
1(t)/dt) relative to the type 2 value-time function (df
2(t)/dt).
• Ratio of Maximum Profits b
1/b
2. We considered the values 1 and 4. When this ratio is 1, both value-time functions have the same maximum values. Depending on the parameter a, this may lead to f
1< f
2for some t. However, when the value is 4, we have f
1> f
2for all t.
We implemented the LP for systems with a maximum queue length of five using AMPL (Fourer
et. al. 1993). We truncated the queue length due to computational limitations. In systems with
a large number of states, each state would have a very small probability, particularly for those
corresponding to long queues. Due to precision limitations in computers, a state with less than
10
−5probability may be rounded to zero and cause its corresponding integer variable z
ito be set
to zero. This would destroy the integrity of LP Constraint (4) and result in an incorrect solution.
Single-Server and Two-Class MDP Model for the Complete Information System
The following MDP represents a single-server system with two job types. It is used in Section 8.1.
• State Space includes states (n
1, n
2, t, q), where n
1and n
2are the number of type 1 and type 2 jobs in the system, respectively, t is the number of time intervals the job under service has been worked on, and q is the type of job under service. The variables n
1, n
2and t are non-negative integers.
• Decision epochs are the beginning of each period.
• Action space includes actions Keep (K), Release and Process Type 1 (R1), and Release and Process Type 2 (R2). Action “Keep” dictates that the worker should continue working on the current job for one more period. Action “Release” implies the worker should stop working on the current job and release it. The worker can then choose between a type 1 and type 2 job for processing if jobs are available. Otherwise, the worker becomes idle.
Letting h
idenote the holding cost per unit time of task type i, we can write the optimality equation of the MDP model for the case where the job in process is type 1 (q = 1), and n
1≥ 2, n
2≥ 1 and t ≥ t
min+ 1 as follows:
δτ g+V (n
1, n
2, t, 1) = max
λδτ
h
γV (n
1+ 1, n
2, t + 1, 1) + (1 − γ)V (n
1, n
2+ 1, t + 1, 1) i
: (K) + (1 − λδτ )V (n
1, n
2, t + 1, 1) − (n
1h
1+ n
2h
2)δτ
λδτ h
γV (n
1, n
2, 1, 1) + (1 − γ)V (n
1− 1, n
2+ 1, 1, 1) i
: (R1) + (1 − λδτ )V (n
1− 1, n
2, 1, 1) −
h
(n
1− 1)h
1+ n
2h
2i
δτ + f
1(t)
λδτ [γV (n
1, n
2, 1, 2) + (1 − γ)V (n
1− 1, n
2+ 1, 1, 2) : (R2) + (1 − λδτ )V (n
1− 1, n
2, 1, 2) −
h
(n
1− 1)h
1+ n
2h
2i
δτ + f
1(t)
where g is the optimal average profit per unit time. The optimality equations for the other three
cases where n
1= 1, n
2≥ 1 and t ≥ t
min+ 1; or n
1≥ 1, n
2= 0 and t ≥ t
min+ 1; or n
1= 1,
n
2= 0 and t ≥ t
min+ 1 can be written in similar fashion. For the cases where t ≤ t
min, optimality
equations can also be written in similar fashion by considering only action Keep. The optimality
equations for the case where the job in process is type 2 (q = 2) follow the same structure, and are
therefore omitted.
Computing the Lower Bound on Value of Information
In this section, we discuss the approach we took to obtain a lower bound of the value of information in DTC systems. It is used in Section 8.1
In order to find the VOI in DTC systems with larger state spaces, we developed an approxi- mation for the partial information case. We did this by estimating the average queue length for each job type using the results from the two-class FCFS M/M/1 queueing model in Gross and Harris (equation 3.45). For each case of our numerical study, we first determined the optimal single-threshold (ST
i) for each job type and use it as the average service time (i.e., 1/µ
i) in order to approximate the average queue length for each job type (L
q,i). Then, we computed the aver- age holding cost for the partial information system as the weighted average of the queue lengths:
h
approx= ¡
h
1L
q,1+ h
2L
q,2)/(L
q,1+ L
q,2). Using this holding cost, we developed a value iteration algorithm for an uncapacitated system under partial information where the state of the system is (n, t, q), and used it to compute the optimal policy for each case of our numerical study. Then, we fed the control policy into the corresponding MDP with the exact holding cost (h
1and h
2) in order to compute the actual profit.
Since this policy is based on the assumption that both jobs have holding cost h
approx, it is suboptimal and therefore produces a lower bound on the profit for the Partial Information system.
Hence, this allows us to compute an upper bound on the VOI in DTC systems.
We conducted the numerical study using the same truncation and convergence requirement used
in Section 4.3. We used the MDP for single-server systems with stochastic value-time function in
Section 7.1, in which each value-time function corresponds to a job type.
Proofs of Analytical Results
LEMMA 1 For t0 ≥ tmin+ 1, if f (t0 + 1) − f (t0) < hδτ , then it is optimal to release in state (n, t) for n ≥ 1 and t ≥ t0.
Proof: In order to prove Lemma 1, we use contradiction. Suppose that the conditions are satisfied in state (n, t) for n ≥ 1 and t ≥ t0, but it is not optimal to release the job. In other words, policy S that keeps the current job until time t + j (for j ≥ 1) and then release it, is optimal. We show that this is not possible.
Consider policy S0 which mimics policy S until time t0. However, at time t0 policy S0 releases the job in state (n, t0) and then remains idle until time t0+ j, while policy S keeps the job in state (n, t0) even though f (t0+1)−f (t0) < hδτ . Therefore, the sample paths under these two polices are the same prior to time t0and after time t0+ j. We define T Psand T Ps0 as the total average profit under policies S and S0, respectively, from time t0 to time t0+ j. If the number of jobs in the system at the beginning of time k for policy S is nk, then
T Ps− T Ps0 =
t0X+j−1 k=t0
−nkhδτ + f (t0+ j) −
t0X+j−1 k=t0
−(nk− 1)hδτ − f (t0)
= −(j − 1)hδτ + f (t0+ j) − f (t0)
= [f (t0+ j) − f (t0+ j − 1) − hδτ ] + [f (t0+ j − 1) − f (t0+ j − 2) − hδτ ]
+ · · · + [f (t0+ 1) − f (t0) − hδτ ] (7)
Since f (t0+ 1) − f (t0) < hδτ and f (t) is concave (i.e., f (t) − f (t − 1) is non-increasing in t), the right hand side of (7) is negative. This means that policy S cannot be optimal, which is a contradiction. Thus, if f (t0+ 1) − f (t0) < hδτ , it is optimal to release in state (n, t) where t ≥ t0.
COROLLARY 1 According to Lemma 1, we can conclude that there exists a Tmax such that Tmax = min{t|f (t + 1) − f (t) < hδτ }, and it is not optimal to work on a job for more than Tmax.
LEMMA 2 The optimal policy in state (1, t) is to keep if tmin< t < Tmax, and to release if t ≥ Tmax. Proof: We first show that policies that release the job in state (1, t) where tmin < t < Tmax are not optimal. Consider policy S that releases the job in state (1, t). Also consider policy S0 which mimics policy S until the first time (t0) policy S releases the job. However, at time t0policy S0keeps the job for one more period until time t0+ 1. During that time period, if there is an arrival, both sample paths reach state (1, 0) at the beginning of time t0+ 1. On the other hand, if there is no arrival, both sample paths reach state (0, 0). Therefore, the sample paths under these two polices are the same before time t0and after time t0+ 1.
We define T Ps and T Ps0 as the total average profit under policies S and S0, respectively, from time t0 to time t0+ 1. Therefore,
T Ps− T Ps0 = f (t) − [−hδτ + f (t + 1)] = hδτ − [f (t + 1) − f (t)]. (8)
Since t < Tmax, we have f (t + 1) − f (t) > hδτ , and so the right hand side of (8) is negative, which implies that policy S cannot be optimal. Repeating the same argument for state (1, t + 1), we conclude that it is not optimal to release the job in state (1, t) for tmin < t < Tmax.
Using the same argument, it can be shown that policies that keep the job in state (1, t) where t ≥ Tmaxare not optimal. Thus, the optimal policy in state (1, t) is to release if t ≥ Tmax.
LEMMA 3 If ρmin < 1, then there exists an optimal stationary policy that maximizes the total average profit per unit time. Furthermore, the gain rate g is constant and the value iteration algorithm converges.
Proof: Before we start the proof of Lemma 3, we first introduce Π as the class of stationary policies that do not work on any job for more than Tmax units of time.
We now show that every stationary policy of our MDP model generates a Markov chain with a single irreducible class. Since it is not optimal to work on a job for more than Tmax, without loss of generality, we focus on the class of stationary policies Π. First, because arrivals to the system are Poisson, state (0, 0) can reach state (1, 1) with a positive probability. Also, according to Lemma 2, state (1, 1) can reach any state (1, t), for 1 ≤ t ≤ Tmax with positive probability. It is also clear that state (1, t) can reach any state (n, t) where n ≥ 1 with a positive probability. Consequently, state (0, 0) can reach any state (n, t) with a positive probability. Similarly, with a positive probability, states (n, t) for all n and t ≤ Tmax can reach state (0, 0). This happens when no arrival occurs during the service of the n jobs. Therefore, the Markov chain corresponding to every stationary policy in the class of Π is irreducible, and thus we can use Theorem 8.10.7 in Puterman (1994) to prove that there exists an average-cost optimal stationary policy with a constant average cost g. Furthermore, since r(s, Keep) = −nhδτ and r(s, Release) = f (t) − nhδτ , we have r(s, a) < ∞ for any state s = (n, t) and action a. Therefore, it suffices to prove that Theorem 8.10.9 in Puterman (1994) holds.
In order to show that this theorem applies, we first need to prove that when λtmin < 1, there exists a stationary policy that induces a positive recurrent Markov chain, and also results in a finite (bounded) average profit. Consider stationary policy θ that works on all jobs for exactly tmin units of time. Under this policy, the system behaves like an M/D/1 queue. Since λtmin < 1, the queue is bounded (i.e., the average holding cost per unit time is bounded). Furthermore, the average value obtained by the M/D/1 system per unit time is also bounded, since it is equal to λf (tmin). Therefore, gθ, the average profit per unit time under policy θ is bounded. Also, since the system under policy θ is an M/D/1 queue, it is easy to show that the underlying Markov chain is positive recurrent. Thus, to verify that all assumptions of Theorem 8.10.9 in Puterman (1994) hold, it only remains to prove that E = {s ∈ S : r(s, a) > gθ} is nonempty and finite. We do this as follows:
It is easy to show that E is a finite and non-empty set:
• E is a finite set: In our MDP model, we have r(s, Keep) = −nhδτ . Furthermore, we have already shown that there exists a stationary policy θ with finite profit gθ. Since r(s, Keep) is decreasing in n,
it is clear that there are only finitely many states s = (n, t) that satisfy r(s, Keep) = −nhδτ > gθ.
• E is a non-empty set: We use contradiction to show that E is non-empty, that is, if E is an empty set, the optimal average profit cannot be g. If E is an empty set, then all states have profit per transition r(s, a) ≤ g, which results in the optimal average profit to be less than or equal to g. Hence, the optimal average profit can only be g if all r(s, a) = g. However, all r(s, a) are different depending on its state s and action a. Therefore, the optimal average profit cannot be g, which is a contradiction.
Thus, since all the assumptions of Theorem 9.10.9 in Puterman (1994) are verified, there exists an average- cost optimal stationary policy where the average reward g is constant.
Finally, we show that the value iteration algorithm converges. Proposition 4.3 of Sennott (1996) states that, if (i) there exists a stationary policy e inducing an irreducible and positive recurrent Markov chain with the finite average cost Je < ∞, and (ii) there exists ² > 0 such that B = {s| there exists a such that r(s, a) >
Je+ ²} is finite, then the value iteration algorithm converges. We have already shown (i). For (ii), using the same argument as we did for set E, it can be easily shown that set B is finite (assume ² = 0.1 for example).
Thus, both conditions of Proposition 4.3 are satisfied, and hence the value iteration algorithm converges.
THEOREM 1: If f (t) < ht for all t, then the optimal policy is to spend the minimum amount of time (i.e., tmin) on each job.
Proof: Consider the profit function Π(t) = f (t) − h t. Given that f (t) is increasing in t, Π(t) is negative and decreasing in t if f (t) < h t for all t. Therefore, in order to maximize profit, it is optimal to release the job as soon as possible. In other words, the optimal processing time is the minimum required (tmin).
LEMMA 3 If it is optimal to keep in state (n, t) for n ≥ 1 and t > tmin, then t < Tmax.
Proof: To prove Lemma 3, we use contradiction. Suppose that it is optimal to keep in state (n, t), but we have t ≥ Tmax. However, according to Corollary 1, if t ≥ Tmax, then the optimal policy is to release, which is a contradiction. Thus, if it is optimal to keep in state (n, t), then t < Tmax.
PROPOSITION 1 If tmin= 0, then it is optimal to release in state (1, 1) if and only if f (t) < ht for all t.
Proof: If f (t) < ht for all t, then the profit gained from a job (regardless of the job processing time) is always greater than the holding cost of keeping the job in the system. Thus, it is not optimal to keep any jobs in the system at any time, including in state (1,1). The reverse can be easily proven using a contradiction argument.
PROPOSITION 2 If f (t) ≥ ht for some t, then the optimality equation has the following properties:
C1: V (n + 1, t) − V (n, 1) is non-increasing in n for n ≥ 1 and t ≥ tmin+ 1 C2: V (n, t) − V (n, t + 1) is non-increasing in n for n ≥ 1 and t ≥ 1 C3: V (n, t) − V (n − 1, t) is non-increasing in n for n ≥ 2 and t ≥ 1 C4a: V (1, t + 1) − V (2, t + 1) + V (1, 1) − V (0, 0) ≥ 0 for t ≥ tmin
C4b: V (2, t + 1) − V (3, t + 1) + V (2, 1) − V (1, 0) ≥ 0 for t ≥ tmin
C5: V (n, t + 1) − V (n, t) ≥ 0 for n ≥ 1 and t ≥ 1
C6: (1 − λδτ )[V (n − 1, 1) − V (n, t + 1)] + λδτ [V (n, 1) − V (n + 1, t + 1)] + f (t) is non-decreasing in t for n ≥ 2 and t ≥ tmin+ 1
Proof: We use induction and the value-iteration algorithm to prove C1 to C6. It is clear that at the first iteration of the value iteration algorithm, Conditions C1 to C6 hold (since V0(n, t) = 0 for all n and t). We assume that Conditions C1 to C6 hold at iteration k, and we will prove that they also hold at iteration k + 1.
PROOF for C1:
We prove that Vk+1(n + 1, t) − Vk+1(n, 1) is non-increasing in n at iteration k + 1 for n ≥ 1 and t ≥ tmin+ 1.
We have for t ≥ tmin+ 1:
Vk+1(n + 1, t) = max
(1 − λδτ )Vk(n + 1, t + 1) + λδτ Vk(n + 2, t + 1) − (n + 1)hδτ : Keep (1 − λδτ )Vk(n, 1) + λδτ Vk(n + 1, 1) + f (t) − nhδτ : Release and
Vk+1(n, 1) = max
(1 − λδτ )Vk(n, 2) + λδτ Vk(n + 1, 2) − nhδτ : Keep
(1 − λδτ )Vk(n − 1, 1) + λδτ Vk(n, 1) + f (1) − (n − 1)hδτ : Release for n ≥ 2 (1 − λδτ )Vk(0, 0) + λδτ Vk(1, 0) + f (1) : Release for n = 1 Since Conditions C1 to C6 hold at iteration k, there are only three possible cases for the optimal actions at states (n + 1, t) and (n, 1). These cases are as follows:
Action at Action at Case state (n + 1, t) state (n, 1)
I Keep Keep
II Release Keep
III Release Release
C1, Case I: In this case the optimal action in both states (n + 1, t) and (n, 1) is Keep. Therefore, we have:
Vk+1(n + 1, t) − Vk+1(n, 1) = (1 − λδτ )[Vk(n + 1, t + 1) − Vk(n, 2)] + λδτ [Vk(n + 2, t + 1) − V (n + 1, 2)] − hδτ
= (1 − λδτ )[Vk(n + 1, t + 1) − Vk(n, 1)] + λδτ [Vk(n + 2, t + 1) − Vk(n + 1, 1)]
+(1 − λδτ )[Vk(n, 1) − Vk(n, 2)] + λδτ [Vk(n + 1, 1) − Vk(n + 1, 2)] − hδτ.
The first two terms on the right hand side are non-increasing in n, since Condition C1 holds at iteration k.
The last two terms are also non-increasing in n due to C2. Thus, Vk+1(n+1, t)−Vk+1(n, 1) is non-increasing in n.
C1, Case II: In this case the optimal actions in states (n+1, t) and (n, 1) are Release and Keep, respectively.
Thus, we have:
Vk+1(n + 1, t) − Vk+1(n, 1) = (1 − λδτ )[Vk(n, 1) − Vk(n, 2)] + λδτ [Vk(n + 1, 1) − V (n + 1, 2)] + f (t).
Since C2 holds at iteration k, then the above equation is non-increasing in n.
C1, Case III: In this case the optimal action in both states (n + 1, t) and (n, 1) is Release. Thus, for n ≥ 2, we have:
Vk+1(n + 1, t) − Vk+1(n, 1) = (1 − λδτ )[Vk(n, 1) − Vk(n − 1, 1)]
+λδτ [Vk(n + 1, 1) − V (n, 1)] − hδτ + f (t) − f (1).
Since C3 holds at iteration k, the above equation is non-increasing in n.
When n = 1, the action in state (n = 1, 1) is Release. However, this case is not considered in Proposition 2, since as shown in Proposition 1, if the optimal action in state (1, 1) is Release, then we will have f (t) < ht for all t. This completes the proof for C1.
PROOF for C2:
We prove that Vk+1(n, t) − Vk+1(n, t + 1) is also non-increasing in n at iteration k + 1 for n ≥ 1 and t ≥ 1.
For state (n, t), when 1 ≤ t ≤ tmin, we have:
Vk+1(n, t) = (1 − λδτ )Vk(n, t + 1) + λδτ Vk(n + 1, t + 1) − nhδτ : Keep When t ≥ tmin+ 1, we have:
Vk+1(n, t) = max
(1 − λδτ )Vk(n, t + 1) + λδτ Vk(n + 1, t + 1) − nhδτ : Keep
(1 − λδτ )Vk(n − 1, 1) + λδτ Vk(n, 1) + f (t) − (n − 1)hδτ : Release for n ≥ 2 (1 − λδτ )Vk(0, 0) + λδτ Vk(1, 0) + f (t) : Release for n = 1 For state (n, t + 1), when 1 ≤ t ≤ tmin− 1, we have:
Vk+1(n, t + 1) = (1 − λδτ )Vk(n, t + 2) + λδτ Vk(n + 1, t + 2) − nhδτ : Keep and, when t ≥ tmin, we have:
Vk+1(n, t+1) = max
(1 − λδτ )Vk(n, t + 2) + λδτ Vk(n + 1, t + 2) − nhδτ : Keep
(1 − λδτ )Vk(n − 1, 1) + λδτ Vk(n, 1) + f (t + 1) − (n − 1)hδτ : Release for n ≥ 2 (1 − λδτ )Vk(0, 0) + λδτ Vk(1, 0) + f (t + 1) : Release for n = 1 Since Conditions C1 to C6 hold at iteration k, there are only three possible cases for the optimal actions in states (n, t) and (n, t + 1). These cases are as follows:
Action at Action at Range for Case state (n, t) state (n, t + 1) t
I Keep Keep t ≥ 1
II Keep Release t ≥ tmin
III Release Release t ≥ tmin+ 1
C2, Case I: In this case, the optimal action in both states (n, t) and (n, t + 1) is Keep. Thus, for t ≥ 1 we have:
Vk+1(n, t) − Vk+1(n, t + 1) = (1 − λδτ )[Vk(n, t + 1) − Vk(n, t + 2)] + λδτ [Vk(n + 1, t + 1) − V (n + 1, t + 2)].
Since C2 holds at iteration k, the above equation is non-increasing in n.
C2, Case II: In this case, the optimal actions in states (n, t) and (n, t+1) are Keep and Release, respectively.
Therefore, for t ≥ tmin and n ≥ 2, we have:
Vk+1(n, t) − Vk+1(n, t + 1) = (1 − λδτ )[Vk(n, t + 1) − Vk(n − 1, 1)]
+ λδτ [Vk(n + 1, t + 1) − V (n, 1)] − f (t + 1) − hδτ.
Since C1 holds at iteration k, the above equation is non-increasing in n.
In order to show that Vk+1(n, t) − Vk+1(n, t + 1) is non-increasing in n when n = 1, we need to prove that:
[Vk+1(1, t) − Vk+1(1, t + 1)] − [Vk+1(2, t) − Vk+1(2, t + 1)] ≥ 0.
Note that in Case I, the optimal actions in states (n = 1, t) and (n = 1, t + 1) are Keep and Release, respectively. Since the optimal action in state (n = 1, t + 1) is Release, the optimal action in state (2, t + 1) is also Release. However, the optimal action in state (2, t) can be either Keep or Release.
• If the optimal action in state (2, t) is Release, then we will have:
[Vk+1(1, t) − Vk+1(1, t + 1)] − [Vk+1(2, t) − Vk+1(2, t + 1)]
= h
(1 − λδτ )[Vk(1, t + 1) − Vk(0, 0)] + λδτ [Vk(2, t + 1) − V (1, 0)]i
− h
(1 − λδτ )[Vk(1, 1) − Vk(1, 1)] + λδτ [Vk(2, 1) − V (2, 1)]
i
−(1 − 1 + 1)hδτ − f (t + 1) − f (t) + f (t + 1)
= (1 − λδτ )[Vk(1, t + 1) − Vk(0, 0)] + λδτ [Vk(2, t + 1) − V (1, 0)] − hδτ − f (t + 1). (9)
Since it is optimal to keep in state (1, t), the profit under action Keep in state (1, t) is greater than that under action Release. Thus, we will have:
(1 − λδτ )[Vk(1, t + 1) − Vk(0, 0)] + λδτ [Vk(2, t + 1) − V (1, 0)] − hδτ − f (t) ≥ 0.
Moreover, we know that, f (t + 1) ≥ f (t). Hence, if the above is true, then Equation (9) is ≥ 0. In other words, Condition C2 holds in this case.
• If the optimal action in state (2, t) is Keep, then we have:
[Vk+1(1, t) − Vk+1(1, t + 1)] − [Vk+1(2, t) − Vk+1(2, t + 1)]
= h
(1 − λδτ )[Vk(1, t + 1) − Vk(0, 0)] + λδτ [Vk(2, t + 1) − V (1, 0)]
i
− h
(1 − λδτ )[Vk(2, t + 1) − Vk(1, 1)] + λδτ [Vk(3, t + 1) − V (2, 1)]
i
−(1 − 2 + 1)hδτ − f (t + 1) + f (t + 1)
The first and second terms on the right hand side are non-negative, since Conditions C4a and C4b hold at iteration k. Thus, C2 also hold in this case.
Considering the above two cases, we conclude that Vk+1(n, t) − Vk+1(n, t + 1) is also non-increasing in n for n = 1.
C2, Case III: In this case, the optimal action in both states (n, t) and (n, t + 1) is Release. Therefore, when n ≥ 2, we have:
Vk+1(n, t) − Vk+1(n, t + 1) = (1 − λδτ )[Vk(n − 1, 1) − Vk(n − 1, 1)] + λδτ [Vk(n, 1) − V (n, 1)] + f (t) − f (t + 1)
= f (t) − f (t + 1),
which is non-increasing in n. When n = 1, since the optimal action in states (n = 1, t) is Release, then according to Lemma 2, t ≥ Tmax. Therefore, the optimal action in all states (1, t + 1), (2, t), and (2, t + 1) is also Release. Therefore,
[Vk+1(1, t) − Vk+1(1, t + 1)] − [Vk+1(2, t) − Vk+1(2, t + 1)]
= h
(1 − λδτ )[Vk(0, 0) − Vk(0, 0)] + λδτ [Vk(1, 0) − V (1, 0)] + f (t) − f (t + 1)i
−h
(1 − λδτ )[Vk(1, 1) − Vk(1, 1)] + λδτ [Vk(2, 1) − V (2, 1)] + f (t) − f (t + 1)i
= 0,
which implies that Vk+1(n, t) − Vk+1(n, t + 1) is also non-increasing in n for n = 1. This completes the proof for C2.
PROOF for C3:
We prove that Vk+1(n, t) − Vk+1(n − 1, t) is non-increasing in n at iteration k + 1 for n ≥ 2 and t ≥ 1. For state (n, t), when 1 ≤ t ≤ tmin, we have:
Vk+1(n, t) = (1 − λδτ )Vk(n, t + 1) + λδτ Vk(n + 1, t + 1) − nhδτ : Keep When t ≥ tmin+ 1, we have:
Vk+1(n, t) = max
(1 − λδτ )Vk(n, t + 1) + λδτ Vk(n + 1, t + 1) − nhδτ : Keep (1 − λδτ )Vk(n − 1, 1) + λδτ Vk(n, 1) + f (t) − (n − 1)hδτ : Release
For state (n − 1, t), when 1 ≤ t ≤ tmin, we have:
Vk+1(n − 1, t) = (1 − λδτ )Vk(n − 1, t + 1) + λδτ Vk(n, t + 1) − (n − 1)hδτ : Keep When t ≥ tmin+ 1, we have:
Vk+1(n−1, t) = max
(1 − λδτ )Vk(n − 1, t + 1) + λδτ Vk(n, t + 1) − (n − 1)hδτ : Keep
(1 − λδτ )Vk(n − 2, 1) + λδτ Vk(n − 1, 1) + f (t) − (n − 2)hδτ : Release for n ≥ 3 (1 − λδτ )Vk(0, 0) + λδτ Vk(1, 0) + f (t) : Release for n = 2 Since Conditions C1 to C6 hold at iteration k, there are only three cases for the optimal actions in states (n, t) and (n − 1, t). These cases are as follows:
Action at Action at Range for Case state (n, t) State (n − 1, t) t
I Keep Keep t ≥ 1
II Release Keep t ≥ tmin+ 1
III Release Release t ≥ tmin+ 1
C3, Case I: In this case, the optimal action in both states (n, t) and (n − 1, t) is Keep. Thus, for t ≥ 1, we have:
Vk+1(n, t) − Vk+1(n − 1, t) = (1 − λδτ )[Vk(n, t + 1) − Vk(n − 1, t + 1)]
+λδτ [Vk(n + 1, t + 1) − V (n, t + 1)] − hδτ Since C3 holds at iteration k, the above equation is non-increasing in n.
C3, Case II: In this case, the optimal actions in states (n, t) and (n−1, t) are Release and Keep, respectively.
Therefore, for t ≥ tmin+ 1, we have:
Vk+1(n, t) − Vk+1(n − 1, t)
= (1 − λδτ ) n
Vk(n − 1, 1) − Vk(n − 1, t + 1) o
+ λδτ n
Vk(n, 1) − V (n, t + 1) o
+ f (t)
= (1 − λδτ ) n
[Vk(n − 1, 1) − Vk(n − 1, 2)] + [Vk(n − 1, 2) − Vk(n − 1, 3)] + · · · + [Vk(n − 1, t) − Vk(n − 1, t + 1)]
o
+λδτ n
[Vk(n, 1) − Vk(n, 2)] + [Vk(n, 2) − Vk(n, 3)] + · · · + [Vk(n, t) − Vk(n, t + 1)]
o + f (t) Since C2 holds at iteration k, then the above equation is non-increasing in n.
C3, Case III: In this case, the optimal action in both states (n, t) and (n − 1, t) is Release. Thus, for t ≥ tmin+ 1, when n ≥ 3, we have:
Vk+1(n, t) − Vk+1(n − 1, t) = (1 − λδτ )[Vk(n − 1, 1) − Vk(n − 2, 1)] + λδτ [Vk(n, 1) − V (n − 1, 1)] − hδτ Since C3 holds at iteration k, the above equation is non-increasing in n. In order to prove that Vk+1(n, t) − Vk+1(n − 1, t) is non-increasing in n for n = 2, we need to show that:
[Vk+1(2, t) − Vk+1(1, t)] − [Vk+1(3, t) − Vk+1(2, t)] ≥ 0.
Since in Case III, the optimal action in state (n − 1 = 1, t) is Release, then according to Lemma 2, we have t ≥ Tmax. Consequently, the optimal action at both states (2, t) and (3, t) is Release. Thus, we have:
[Vk+1(2, t) − Vk+1(1, t)] − [Vk+1(3, t) − Vk+1(2, t)]
= h
(1 − λδτ )[Vk(1, t) − Vk(0, 0)] + λδτ [Vk(2, t) − V (1, 0)] + f (t) − f (t) − hδτi
−h
(1 − λδτ )[Vk(2, 1) − Vk(1, 1)] + λδτ [Vk(3, 1) − V (2, 1)] + f (t) − f (t) − hδτi
= (1 − λδτ )h
Vk(1, t) − Vk(2, 1) + Vk(1, 1) − Vk(0, 0)i + λδτ
h
Vk(2, t) − Vk(3, 1) + V (2, 1) − V (1, 0) i
The first and second terms are positive, since Conditions C4a and C4b hold at iteration k. Thus, the entire right hand side is positive. Thus, Vk+1(n, t) − Vk+1(n − 1, t) is also non-increasing in n for n = 2. This completes the proof for C3.
PROOF for C4a:
We prove that Vk+1(1, t + 1) − Vk+1(2, t + 1) + Vk+1(1, 1) − Vk+1(0, 0) is ≥ 0. For t ≥ tmin, we have:
Vk+1(1, t + 1) = max
(1 − λδτ )Vk(1, t + 2) + λδτ Vk(2, t + 2) − hδτ : Keep (1 − λδτ )Vk(0, 0) + λδτ Vk(1, 0) + f (t + 1) : Release
Vk+1(2, t + 1) = max
(1 − λδτ )Vk(2, t + 2) + λδτ Vk(3, t + 2) − 2hδτ : Keep (1 − λδτ )Vk(1, 1) + λδτ Vk(2, 1) + f (t + 1) − hδτ : Release
Vk+1(1, 1) = (1 − λδτ )Vk(1, 2) + λδτ Vk(2, 2) − hδτ : Keep and
Vk+1(0, 0) = (1 − λδτ )Vk(0, 0) + λδτ Vk(1, 0) : Keep
Since Conditions C1 to C6 hold at iteration k, there are only three cases for the optimal actions in states (1, t + 1), (2, t + 1), (1, 1), and (0, 0). These cases are as follows:
Action at Action at Action at Action at Case state (1, t + 1) state (2, t + 1) state (1, 1) state (0, 0)
I Keep Keep Keep Keep
II Keep Release Keep Keep
III Release Release Keep Keep
C4a, Case I: In this case, the optimal action at all four states is Keep. Thus, for t ≥ tmin, we have:
Vk+1(1, t + 1) − Vk+1(2, t + 1) + Vk+1(1, 1) − Vk+1(0, 0)
= (1 − λδτ ) h
Vk(1, t + 2) − Vk(2, t + 2) + Vk(1, 2) − Vk(0, 0) i
+ λδτ h
Vk(2, t + 2) − V (3, t + 2) + Vk(2, 2) − V (1, 0) i
+ hδτ (−1 + 2 − 1)
= (1 − λδτ ) h
Vk(1, t + 2) − Vk(2, t + 2) + Vk(1, 1) − Vk(0, 0) i
(10) + λδτh
Vk(2, t + 2) − V (3, t + 2) + Vk(2, 1) − V (1, 0)i
(11) +(1 − λδτ )h
Vk(1, 2) − Vk(1, 1)] + λδτ [Vk(2, 2) − V (2, 1)i
(12) The first and the second terms of the right hand side are non-negative, since Conditions C4a and C4b hold at iteration k. The last two terms on the above right hand side are also non-negative due to induction assumption C5. Thus, Condition C4a holds in Case I.
C4a, Case II: In this case, the optimal action in states (1, t + 1), (1, 1), and (0, 0) is Keep, while the optimal action in state (2, t + 1) is Release. Thus, for t ≥ tmin, we have:
Vk+1(1, t + 1) − Vk+1(2, t + 1) + Vk+1(1, 1) − Vk+1(0, 0)
= (1 − λδτ ) h
Vk(1, t + 2) − Vk(1, 1) + Vk(1, 2) − Vk(0, 0) i
+ λδτ h
Vk(2, t + 2) − V (2, 1) + Vk(2, 2) − V (1, 0) i
− f (t + 1) + hδτ (−1 + 1 − 1)
= (1 − λδτ )[Vk(1, t + 2) − Vk(0, 0)] + λδτ [Vk(2, t + 2) − V (1, 0)] − f (t + 1) − hδτ (13) +(1 − λδτ )[Vk(1, 2) − Vk(1, 1)] + λδτ [Vk(2, 2) − V (2, 1)] (14) Note that the term (13) on the right hand side is the difference in profit when action Keep is used instead of Release in state (1, t + 1). However, in Case II, the optimal action in state (1, t + 1) is Keep; thus, that term is non-negative. The term (14) on the right hand side is also non-negative due to induction assumption C5. Thus, Condition C4a holds in Case II.
C4a, Case III: In this case, the optimal action in states (1, t + 1) and (2, t + 1) is Release, while the optimal action in states (1, 1), and (0, 0) is Keep. Thus, for t ≥ tmin, we have: For case IV, when t ≥ tmin
Vk+1(1, t + 1) − Vk+1(2, t + 1) + Vk+1(1, 1) − Vk+1(0, 0)
= (1 − λδτ )h
Vk(0, 0) − Vk(1, 1) + Vk(1, 2) − Vk(0, 0)i + λδτ
h
Vk(1, 0) − V (2, 1) + Vk(2, 2) − V (1, 0) i
+ f (t + 1) − f (t + 1) + hδτ (1 − 1)
= (1 − λδτ )[Vk(1, 2) − Vk(1, 1)] + λδτ [Vk(2, 2) − Vk(2, 1)]
Since C5 holds at iteration k, then the right hand side is non-negative. Thus, Condition C4a holds in Case III. This completes the proof for C4a.
PROOF for C4b:
We prove that Vk+1(2, t + 1) − Vk+1(3, t + 1) + Vk+1(1, 1) − Vk+1(2, 1) ≥ 0. For t ≥ tmin, we have:
Vk+1(2, t + 1) = max
(1 − λδτ )Vk(2, t + 2) + λδτ Vk(3, t + 2) − 2hδτ : Keep (1 − λδτ )Vk(1, 1) + λδτ Vk(2, 1) + f (t + 1) − hδτ : Release
Vk+1(3, t + 1) = max
(1 − λδτ )Vk(3, t + 2) + λδτ Vk(4, t + 2) − 3hδτ : Keep (1 − λδτ )Vk(2, 1) + λδτ Vk(3, 1) + f (t + 1) − 2hδτ : Release
Vk+1(2, 1) = max
(1 − λδτ )Vk(2, 2) + λδτ Vk(3, 2) − 2hδτ : Keep (1 − λδτ )Vk(1, 1) + λδτ Vk(2, 1) + f (1) − hδτ : Release and
Vk+1(1, 0) = (1 − λδτ )Vk(1, 1) + λδτ Vk(2, 1) − hδτ : Keep only
Since Conditions C1 to C6 hold at iteration k, there are only four possible cases for the optimal actions in states (2, t + 1), (3, t + 1), (2, 1) and (1, 0). These case are:
Action at Action at Action at Action at Case state (2, t + 1) state (3, t + 1) state (2, 1) state (2, 1)
I Keep Keep Keep Keep
II Keep Release Keep Keep
III Release Release Keep Keep
IV Release Release Release Keep
C4b, Case I: For case I, when t ≥ tmin
Vk+1(2, t + 1) − Vk+1(3, t + 1) + Vk+1(2, 1) − Vk+1(1, 0)
= (1 − λδτ )[Vk(2, t + 2) − Vk(3, t + 2) + Vk(2, 2) − Vk(1, 1)]
+λδτ [Vk(3, t + 2) − V (4, t + 2) + Vk(3, 2) − V (2, 1)] + hδτ (−2 + 3 − 1 + 1)
= (1 − λδτ )[Vk(2, t + 2) − Vk(1, 1) − Vk(3, t + 2) + Vk(2, 1)] (15) +λδτ [Vk(3, t + 2) − V (2, 1) − Vk(4, t + 2) + V (3, 1)] (16)
+ (1 − λδτ )[Vk(2, 2) − Vk(2, 1)] (17)
+λδτ [V (3, 2) − Vk(3, 1)] + hδτ (18)
From C1, we know that (15) and (16) are ≥ 0. (17) and (18) are also ≥ 0 from C5.
C4b, Case II: In this case, the optimal action in states (2, t+1), (2, 1), and (2, 1) is Keep, while the optimal action in state (3, t + 1) is Release. Thus, for t ≥ tmin, we have:
