Approximate Dynamic Programming (ADP) Approach

Due to the uncertainty and stochastic nature of natural gas storage decisions, scholars such as Lai et al. (2008), gravitated toward stochastic dynamic programming (SDP) as the natural approach to solving the storage valuation problem. Solving dynamic optimization problems is rooted in the application of Bellman’s equation which solves simpler sub-problems, computing the value of a decision at a certain point in time in terms of the payoff of the current decision and the value of the remaining decisions within the problem that result from the prior decisions. Bellman’s equation for stochastic problems can be described as

𝑉𝑡 𝑠𝑡, 𝑎𝑡 = max 𝐶 𝑥, 𝑎 + 𝛾𝔼 𝑉𝑡+1 𝑠𝑡+1, 𝑎𝑡+1 |𝑠𝑡, 𝑎𝑡 ∀ 𝑡 ∈ 𝑇,

where C(x, a) is the contribution made at time t from taking action a while in state x, and γ discounts the expected value of the next state (Puterman 1994).

Storage value is determined by the expected cash flow generated by the operational decisions. These decisions are dependent on the current operational characteristics of the storage facility in addition to the market conditions at the

CHAPTER 2: Literature Review

26

time. Given the decision to inject or withdraw x and the operational and market characteristics that make up the state at time t, the value of storage can be expressed by the following Bellman’s equation

𝑉_𝑡 𝑋𝑡, 𝑭𝑡 = arg max 𝐶𝐹𝑡 𝑥, 𝑆𝑡 + 𝑒−𝑟𝑑𝑡𝔼 𝑉𝑡+1 𝑋𝑡+1, 𝑭𝑡+1 |𝑭𝑡 ∀ 𝑡 ∈ 𝑇, subject to −𝑊𝑡 ≤ 𝑥 ≤ 𝐼𝑡, 𝑋_𝑡+1 = 𝑥 + 𝑋_𝑡, 0 ≤ 𝑥 ≤ 𝑋_𝑚𝑎𝑥, 𝐶𝐹_𝑡 𝑓, 𝑆𝑡 = 𝑓 ∗ 𝑆𝑡, where

𝑋𝑡 = the inventory level at time t with 𝑋𝑚𝑎𝑥being total capacity

𝑥 = the amount of flow in any given period t

𝑊_𝑡 = the maximum withdrawal capacity at time t

𝐼_𝑡 = the maximum injection capacity at time t

𝑠𝑡 = the gas price at time t which is determined by a specific stochastic process

𝑭𝑡 = the forward curve at time t

CHAPTER 2: Literature Review

27

The solution approach for solving a discrete SDP is defined by backward induction, which can be implemented by first considering the last time a decision is made and choosing what to do in any situation at that time. Using this information, one can then determine what to do at the second-to-last time of decision. This process continues backwards until one has determined the best action for every possible situation.

Though it appears an easily obtained solution can be reached, the presence of the forward curve vector, 𝑭𝑡, within system state that complicates the ability to

reach a solution. 𝑭𝑡 is a vector of forward prices at time t for all the futures contracts

that mature at various times in the future. Over 72 contracts are traded in the futures market at one time, yet utilizing even a fraction of them within the state variable subjects the problem to the “curse of dimensionality.” This means that the high dimensionality of the state space and the manipulation of exponentially large volumes of information render the solution unfeasible by backwards induction.

To combat this problem Lai et al. (2008) develops a technique based on ADP methods to value the storage of natural gas and renders the high-dimensional model of the forward curve more pliable. This approach transforms the intractable stochastic dynamic program model of the storage problem into a manageable lower dimensional Markov decision process.

The approach to reducing the computationally intractable SDP model is to develop an approximate model using information reduction. Lai et al. (2008), removes price related state variables from the state definition of the exact model.

CHAPTER 2: Literature Review

28

Though the amount of variables removed may fluctuate based on the solver’s discretion, it is important to note that the more price-related variables are removed, the easier it will be to solve efficiently. The contraction within the ADP technique occurs by computing the optimal value function by conditioning on the possible values of the next month’s futures price and the next months futures price at 𝑡 = 0, instead of conditioning on the whole forward curve. So now the vector 𝑭_𝑡, is composed of two scalar variables, 𝑭_𝑡,𝑡+1 and 𝑭_0,𝑡+1. The state variable now includes the current inventory, spot price, and only two variables which are associated with the forward curve within each period t. Thus, the ADP model of the problem is vastly reduced from the previous model and is given by

𝑉_𝑡𝐴𝐷𝑃 _𝑋

𝑡, 𝑭𝑡 = 𝔼 𝑣𝑡𝐴𝐷𝑃 𝑋𝑡, 𝑭𝑡,𝑡+1 |𝑭0,𝑡+1 ∀ 𝑡 ∈ 𝑇,

where,

𝑣_𝑡𝐴𝐷𝑃 _𝑋

𝑡, 𝑭𝑡 = arg max 𝐶𝐹𝑡 𝑥, 𝑆𝑡 + 𝑒−𝑟𝑑𝑡𝔼 𝑉𝑡+1𝐴𝐷𝑃 𝑋𝑡+1, 𝑆 𝑡+1 |𝑭𝑡,𝑡+1 .

The conditioning on the next month’s futures price at time 0, 𝑭0,𝑡+1, can be

seen in the first equation and yields the expected value of the value function displayed by the second equation, which is conditioned on the next month’s futures price at time t, 𝑭𝑡,𝑡+1.

It is reported that the implementation of this ADP model can generate values that on average are 97% of optimal storage value, while the LP model of Section 2.1 reports a larger underestimate of the value of storage with an average of 75% (Lai, Margot and Secomandi 2008). It appears that ADP dominates the LP model in terms

CHAPTER 2: Literature Review

29

of the actual valuation of the storage facility. However, looking at cpu run time as an alternative indicator, ADP can be considered as a suboptimal policy with extensively higher cpu requirements, averaging 250 cpu seconds compared to the .0038 cpu seconds required by the LP model. It is evident that despite the improvements in the actual valuation of natural gas storage, greater computational burdens are undertaken to produce such results. The fact that ADP solution methods are considered slower and less efficient, little impression is made on industry practitioners. Thus, it is important to find a method that will best balance valuation quality and computational efficiency.

Chapter 3 will discuss the stochastic optimization method known as optimal learning, focusing on the stochastic search method known as the Knowledge Gradient Policy. We will utilize this process in order to find a valuation method for natural gas storage that deploys stochastic optimization methods lacking in the LP model, yet will yield less computational burdens found in the ADP model.