Supplement to “A divide and conquer algorithm for exploiting
policy function monotonicity”
(Quantitative Economics, Vol. 9, No. 2, July 2018, 521–540)
GreyGordon
Department of Economics, Indiana University ShiQiu
Department of Economics, Indiana University
AppendixC gives additional sufficient conditions for monotonicity and shows how the results may be applied. AppendixDshows the algorithms deliver cor-rect solutions and incidentally shows how additional state variables (for which one does not want to exploit monotonicity) may be handled. AppendixE quan-titatively evaluates binary monotonicity with sorting while also showing (1) how binary monotonicity with sorting can be used to solve some problems that do not immediately fit the class of nonmonotone problems and (2) how choice variables for which one does not want to (or cannot) exploit monotonicity may be handled. AppendixFcontains all the omitted proofs and lemmas.
AppendixC: Additional sufficient conditions and applications
This Appendix expands on the discussion of monotonicity sufficient conditions in Sec-tion 5 in two ways. First, SecSec-tionC.1gives sufficient conditions for choice correspon-dences to be ascending and for functions to exhibit increasing differences. Second, Sec-tionC.2applies these sufficient conditions and the ones from the main text to establish monotonicity in the RBC model, theArellano(2008) model, and the sorted problem (6). AppendixFgives the proofs.
C.1 Ascending correspondences and increasing differences
Proposition 3 requires the choice correspondence to be ascending. One way to ensure this is for every choice to be feasible. The following lemma, which relies on increasing differences, provides an alternative.
Lemma1. LetII⊂RandI:I→P(I).SupposeI(i)= {i∈I|hm(i i)≥0for allm∈ M}withMarbitrary.IfIis increasing,hmis decreasing ini,andhmhas increasing dif-ferences onI×I(for allm),thenIis ascending onI.
Grey Gordon:[email protected]
Shi Qiu:[email protected]
For an example application of Lemma 1, consider the RBC model and define c(k k z):= −k+zF(k)+(1−δ)k. Since c is increasing in k and decreasing in k, the budget constraint{k∈K|c(k k z)≥0}will be ascending onKfor eachzas long as chas increasing differences ink,k.
To apply the sufficient conditions Proposition 3 and Proposition 4 or to apply Lemma 1, one must establish functions have increasing differences. The following lemma provides a number of sufficient conditions for establishing that this is the case. Some additional conditions may be found inTopkis(1978) andSimchi-Levi, Chen, and Bramel(2014).
Lemma2. ForII⊂RandS⊂I×I,f:S→Rhas increasing differences onSif any of the following hold:
(a) f (i i)=p(i)+q(i)for arbitrarypandq.
(b) f (i i)=p(i)+q(i)+r(i)s(i)for arbitrarypandqwithrandsboth increasing or decreasing.
(c) f (i i)agrees withg:L⊂R2→R,aC2function havingg12≥0andLa hypercube
withS⊂L.
(d) f (i i)is a nonnegative linear combination(i.e.,f =αkfkwithαk≥0)of func-tions having increasing differences.
(e) f (i i)=h(g(i i))forhan increasing,convex,C2function andgincreasing(ini andi)and having increasing differences.
(f ) f (i i)=h(g(i i))forhan increasing,concave,C2function andgincreasing ini, decreasing ini,and having increasing differences.
(g) f (i i)=Eg(h(i ε) i) dF(ε)withghaving increasing differences on{(h iˆ )| ˆh= h(i ε) ε∈E (i i)∈S}andhincreasing ini.
(h) f (i i)=maxx∈Γ (ii)g(i i x) exists for all(i i)∈S,S is a lattice, Γ :S→P(X), X⊂R,the graph ofΓ is a lattice,andghas increasing differences ini,iandi,xandi,x onI×I,I×X,andI×X,respectively(for alli,i,x).
Since our method primarily exploits monotonicity of one choice variable, it will be convenient in some cases to construct an indirect utility function over the other choices and then establish that the indirect utility function has increasing differences. Lemma2
part (h) gives one sufficient condition for this, but its conditions can be difficult to guar-antee unless every choice is feasible. Proposition 5 provides an alternative sufficient condition that may be easier to verify.
Proposition 5. LetS⊂R2be open and convex.Let f :S→Rbe defined byf (i i)=
maxx∈Xu(g(x i i) x)whereuis differentiable,increasing,and concave in its first argu-ment andXis arbitrary.Thenf has increasing differences onSif:
1. fis well defined andC1inion the closure ofS,and
2. for any optimal policyx∗and any(i i)∈S,g2(x∗(i i) i i)exists,is positive,and is
C.2 Applying the sufficient conditions
We now show how the preceding results can be applied to establish monotonicity in the RBC model,Arellano(2008), and the sorted problem (6).
C.2.1 Monotonicity inkin the RBC model Consider the RBC model where we defined c(k k z)= −k+zF(k)+(1−δ)k. By Lemma2part (a),chas increasing differences in k,k. Thenu(c(k k z))andu(c(k k z))+βEz|zV0(k z)have increasing differences
by parts (f ) and (d), respectively. Consequently, an application of Proposition 3 (not-ing the budget constraint is ascend(not-ing by Lemma1) gives thatarg maxku(c(k k z))+ βEz|zV0(k z)is ascending onK. As stated in the main text and proven in AppendixD,
this means binary monotonicity can be used to compute an optimal policyk(k z)that is monotone ink.
C.2.2 Monotonicity inzin the RBC model With additional assumptions, one can also use these results to establishkis monotone inz, although doing so is more complicated. Identical arguments to the above give the optimal choice correspondence as ascending inz if Ez|zV0(k z)has increasing differences in k, z. By Lemma2part (g), this will
hold as long aszis increasing inzandV0(k z)has increasing differences ink,z. To
en-sure this is the case at each step of the Bellman update (assuming the initial guess has increasing differences), one can use Lemma2part (h) if the graph of the choice corre-spondence is a lattice andu◦c+βEz|zV0has increasing differences ink z. A sufficient
condition for the former is that every choice is feasible. A sufficient condition for the lat-ter, by Lemma2part (c), is that∂2u(c)/(∂k ∂z)≥0, which is the same condition required inHopenhayn and Prescott(1992).
C.2.3 Monotonicity inkin the RBC model with elastic labor supply One can establish monotonicity ofk(k z)inkfor the RBC model with elastic labor supply by using Propo-sition5to establish increasing differences of theindirectutility function followed by an application of Proposition 3. Specifically, the maximization problem for a givenz can be written asmaxkU(k k)+βEz|zV (k z)whereU(k k):=maxl∈[01]u(c(l k k) l)
andc(l k k):=max{0−k+zF(k l)+(1−δ)k}. IfU is differentiable and solutions are interior—that is,l∗∈(01)andc(l∗ k k) >0—thenc2(l∗ k k)=zFk(k l∗)+1−δ exists, is positive, and is weakly increasing ink. If in addition consumption is a normal good so thatc(l∗ k k)is decreasing ink, then Proposition5givesUas having increas-ing differences. In this case, Proposition 3 gives monotonicity ofk(k z)ink.
AppendixD: Algorithm correctness for a more general problem
This appendix shows binary monotonicity and the other algorithms deliver correct so-lutions in a more general formulation of (1). SectionD.1gives the more general formu-lation and states the correctness result. SectionD.2shows how the result applies in the RBC andArellano(2008) models while incidentally showing how additional state vari-ables, for which one does not want to exploit monotonicity, may be handled.
D.1 A more general formulation
In both the RBC andArellano(2008) model, there is no guarantee that every choice is feasible. Consequently, one cannot directly use binary monotonicity because the maxi-mization problems do not directly fit (1). To handle this issue in a general way, suppose that the feasible choice set isI(i)⊂ {1 n}, which may be empty, and that the objec-tive function is given by someπ(i i˜ )only defined for(i i)such thati∈I(i). For every isuch thatI(i)is nonempty, define
˜
Π(i)= max
i∈I(i)π˜
i i (7)
Letπdenote a lower bound onπ,˜ 7and define
πi i=
⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩
˜
πi i ifI(i) = ∅andi∈I(i) π ifI(i) = ∅andi∈/I(i) 1i=1 ifI(i)= ∅
(8)
Further, formalize the notion of concavity in the following way.
Definition3. Theproblem is concaveif, for allisuch thatI(i) = ∅,I(i)= {1 n(i)} for some monotone increasing functionn(i)andπ(i·)is either first strictly increasing and then weakly decreasing; or is always weakly decreasing; or is always strictly increas-ing (where defined).
Now we can state the correctness result.
Proposition 6. If I is increasing, then any of brute force, simple, or binary mono-tonicity combined with any of brute force, simple, or binary concavity applied to (1) with the objective function defined as in(8)delivers an optimal solution to(7)provided
arg maxi∈I(i)π(i i˜ )is ascending and the problem is concave as required by the algorithm
choices.
For the proof, see AppendixF.
7A theoretical lower bound onπ˜ is−1+min
D.2 Examples
To see how this result may be applied and how the RBC model can be cast into (7), con-sider the problem’s Bellman update,
V (k z)= max
c≥0k∈Ku(c)+βEz|zV0
k z
s.t.c+k=zF(k)+(1−δ)k
(9)
fork∈K,z∈ZwhereK= {k1 kn}with thekiincreasing. (While here we have used
inelastic labor supply, elastic labor supply can be incorporated by replacing the period utility function with an an indirect utility function as we discuss in AppendixC.) Now, create a separate problem for eachzand write
˜
Πz(i)= max
i∈Iz(i) ˜
πzi i (10)
whereπz˜ andIz are defined as
˜
πzi i:=u−ki+zF(ki)+(1−δ)ki
+βEz|zV0
ki z
Iz(i):=
i∈ {1 n}|ki≤zF(ki)+(1−δ)ki (11)
Then (10) is just (9) withkand k given by grid indices. Moreover, (10) has the same form as (7). Further, becauseIz has the form {1 n¯z(i)}for an increasing function
¯
nz(i), Proposition6shows the monotonicity and concavity algorithms will deliver cor-rect solutions.
TheArellano(2008) model can be mapped into (7) in the same fashion. The main computational difficulty in that model is solving the sovereign’s problem conditional on not defaulting. Specifically, the problem is to solve, for eachb∈Bandy∈Y,
Vn(b y)= max
c≥0b∈Bu(c)+βEy|ymax
Vnb y Vdy
s.t.c+qb yb=b+y
(12)
wherebis the sovereign’s outstanding bonds,q(b y)is the bond price,yis output, and Vdis the value of defaulting. TakingBas{b1 bn}with thebiincreasing and creating
a separate problem for eachy, one has
˜
Πy(i)= max
i∈Iy(i) ˜
πyi i (13)
whereπy˜ andIy are defined as
˜
πyi i:=u−q(bi y)bi+bi+y+βEy|ymax
Vnbi y Vdy
Iy(i):=i∈ {1 n}| −q(bi y)bi+bi+y≥0 (14)
AppendixE: Additional results for the class of nonmonotone problems This appendix builds on Section 4 by testing the quantitative performance of binary monotonicity with sorting. The application, a sovereign default model with endogenous capital accumulation, is described in SectionE.1. SectionE.2assesses the algorithm’s performance inclusive of sorting costs. Last, SectionE.3shows how binary monotonic-ity with sorting can be used to solve some problems that do not fit (6) by transforming them into two-stage problems. It also illustrates how choice variables for which one does not want to (or cannot) exploit monotonicity may be handled.
E.1 A sovereign default model with capital
The model is similar toBai and Zhang(2012) but lacks capital adjustment costs (in Sec-tionE.3we use adjustment costs to illustrate how two-stage reformulations allow bi-nary monotonicity to be used). A sovereign has total factor productivityathat is Markov and chooses bondsband capitalk from setsBandK, respectively, with0∈B. If the sovereign defaults, outputakαfalls by a fractionκ. In equilibrium, the discount bond priceqsatisfiesq(b k a)=(1+r)−1E
a|a(1−d(b k a))whereris an exogenous risk-free rate anddgives the default decision. The sovereign’s problem is to solve
V (b k a)= max
d∈{01}dV
d(k a)+(1−d)Vn(b k a) (15)
where the value of defaulting is
Vd(k a)= max
c≥0k∈Ku(c)+βEa|a
θVdk a+(1−θ)Vn0 k a
s.t.c+k=(1−κ)akα+(1−δ)k
(16)
and the value of repaying is
Vn(b k a)= max
c≥0b∈Bk∈Ku(c)+βEa|aV
b k a
s.t.c+qb k ab+k=akα+(1−δ)k+b
(17)
The most difficult part of computing this model is solving forVn. Note the optimal policies for the problem are generallynotmonotone: An increase in bondsbor capital kmay cause a substitution frombintok or vice versa. This is true even if one uses a cash-at-hand formulation. Nevertheless, binary monotonicity can be used to solve this problem by mapping it into (5) and then sorting to arrive at (6). Specifically, suppose the states(b k)and choices(b k)both lie in a setX= {(bi ki)}having cardinalityn. Creating a separate problem for eacha, one may then write
Van(i)= max
c≥0i∈{1 n}u(c)+Wa
i
s.t.c=za(i)−wai
(18)
Table3. Run times and evaluations for the combinedArellano(2008) and RBC model.
Run times Evaluations per state
Points (m) Brute Simple Binary Brute Simple Binary Speedup Original formulation
50 173 (m) 371 (s) 264 (s) 2500 1273 14 141
100 264 (m) 948 (m) 935 (s) 10,000 5100 16 608 250 161∗(h) 583∗(h) 110 (m) 62,500∗ 31,936∗ 19 3179∗ 500 103∗(d) 372∗(d) 502 (m) 250,000∗ 127,919∗ 21 10668∗ 1000 157∗(d) 570∗(d) 211 (m) 1,000,000∗ 512,379∗ 23 38905∗
Cash-at-hand formulation
50 314 (s) 121 (s) 085 (s) 2500 1304 379 14
100 175 (s) 803 (s) 416 (s) 10,000 5127 860 19 250 412 (m) 176 (m) 310 (s) 62,500 31,723 2467 34 500 323 (m) 128 (m) 227 (m) 250,000 126,354 5432 56 1000 423∗(h) 155∗(h) 101 (m) 1,000,000∗ 503,277∗ 11,855 92∗
Note: Run times are in seconds (s), minutes (m), hours (h), or days (d). The last column gives the run time for simple relative to binary; an∗means the value is estimated; times and average evaluations are over the first200value function iterations.
E.2 Performance
As stated in the main text, binary monotonicity with sorting isO(nlogn)+O(nlogn) as eithernorngrow. While the cost depends only on the total number of pointsnand n, in the case of tensor grids with a fixed number of pointsmalong each dimension, n=md andn=md grow quickly inmwhendandd(the dimensionality of states and choices, resp.) are bigger than1. The cost in these terms isO(mmax{dd}logm)for binary monotonicity andO(md+d)for brute force. While theoretically this results in a massive improvement whend=d>1, the extreme cost of using brute force in this case means one would almost surely reformulate the problem in terms of cash-at-hand, effectively reducingdto1.
Table3reports the run times and evaluation counts for brute force, simple mono-tonicity, and binary monotonicity for different grid sizesm.8In the top panel, the cash-at-hand reformulation has not been used, and so binary monotonicity vastly outper-forms the other methods. For50points in each dimension, binary monotonicity is al-ready14 times faster than simple monotonicity and39times faster than brute force. For a1000points, simple monotonicity’s estimated run time is2months while binary monotonicity’s actual run time is only21minutes. A doubling of the grid sizes makes the speedup increase by roughly a factor of4, which agrees with binary monotonicity beingO(m2logm)and brute force beingO(m4). The speedups measured in evaluation counts are even more dramatic as they exclude time spent sorting.
Reformulating the problem using cash-at-hand (withmpoints for the cash-at-hand state variable) makes brute force an O(m3) algorithm but has no change on binary
8For this example, productivity follows anAR(1)with a persistence parameter of0945and standard
monotonicity’s asymptotics (which are stillO(m2logm)). Consequently, binary mono-tonicity still has an advantage, but it is much smaller. This can be seen in a comparison of the top and bottom panels of Table3. With the cash-at-hand formulation, brute force and simple monotonicity are faster by a factor of roughlym, but binary monotonicity is only twice as fast. Overall, binary monotonicity still outperforms simple monotonic-ity, but by a more modest factor of14to92for run times. The better evaluation count speedups, which are in the34to425range, show that sorting costs are playing a non-trivial role. When onlynorngrow, sorting costs dominate, and here that is essentially the case becausen=m2grows much faster thann=m. However, the speedups mea-sured against brute force—which may be a better benchmark since the sorting of con-tinuation utilities is, to our knowledge, novel—are roughly twice as large.
E.3 Two-stage reformulations
Some models, such as models with adjustment costs, do not directly have the additive separability in the budget constraint of (6). However, they might when breaking the max-imization problem into two stages. For instance, adding capital adjustment costs to our example results in a budget constraint
c+qb k ab+k+ξk−k2=akα+(1−δ)k+b (19)
which can be written asc=za(b k k)−wa(b k k)whereza(b k k):=akα+(1−
δ)k+bandwa(b k k):=q(b k a)b+k+ξ(k−k)2. Consequently, the problem has the form
Van(i j)= max
c≥0iju(c)+Wa
i j
s.t.c=zai j j−wai j j
(20)
so that there is additive separability between theiandi variablesconditional on aj, j pair. Binary monotonicity can be used to solve (20) by breaking it into a two-stage problem wherejis chosen in the first stage andiin the second:
Van(i j)=max
j ˜
Vani j j
˜
Vani j j=max
c≥0iu(c)+Wa
i j
s.t.c=zai j j−wai j j
(21)
For eachj,jcombination, one can sortz(· j j)andW (· j)so that the optimal policy of the second-stage problem is monotone ini. For grids of sizemin each dimension, binary monotonicity with sorting can be used to solve for V˜an inO(m3logm)operations and thenVn
a inO(m3)operations. So, the total cost isO(m3logm), which compares favorably with brute force’sO(m4).
illustrates one way to do this: First, choosejand make it a state variable when choos-ingi. The other way is to first chooseiand make it a state variable when choosingj. The RBC model with elastic labor supply provides an example of this latter approach, and one may consult SectionC.2.3of AppendixCfor more details.
AppendixF: Omitted proofs and lemmas
This appendix gives omitted proofs and lemmas. Section F.1gives the results for the monotonicity-related sufficient conditions. SectionF.2gives proofs and lemmas show-ing binary monotonicity and the other algorithms work correctly. SectionF.3gives the proofs and lemmas for the cost bounds in established in Propositions 1 and 2 in the main text.
F.1 Monotonicity sufficient condition proofs and lemmas
To give the omitted sufficient condition proofs, we first give some definitions and a lemma.
F.1.1 Definitions and a lemma Lattices are general mathematical structures. For our purposes, we need only lattices consisting of subsets of Euclidean space with the component-wise ordering, that is,x≤yforx y∈Rnifxj≤yj for alljwherezj denotes thejth component ofz. In this context, the join operation∨gives the component-wise maximum, namely,x∨y=(max{x1 y1} max{xn yn}). Likewise, the meet operation
∧gives the component-wise minimum,x∧y=(min{x1 y1} min{xn yn}). A lattice
consists of a setX⊂Rn with the component-wise ordering such thatx y∈X implies x∨y x∧y∈X. Note that ifX⊂R, it constitutes a lattice with our ordering:x y∈X impliesmin{x y}max{x y} ∈X. A functionf :X→RwhereX is a lattice is said to be supermodular (submodular) iff (x)+f (y)≤(≥)f (x∧y)+f (x∨y)for all x y∈X; if the inequality is strict for allxandythat cannot be ordered, then the function is strictly supermodular (submodular).
For part of Lemma2, we will need a slightly broader definition of increasing differ-ences than what was given in the main text.
Definition4. LetX⊂Rn. Use the notation(x−ij yi yj)to denote the vectorxbut with theith andjth component replaced with theith andjth component ofy, respectively. A functionf:X→Rhasincreasing differences onXif for alli,jwithi =jand for allyi,yj havingxi≤yiandxj≤yj(such that(x−ij yi xj) (x−ij xi xj) (x−ij yi yj) (x−ij xi yj)∈ X) one has
f (x−ij yi xj)−f (x−ij xi xj)≤f (x−ij yi yj)−f (x−ij xi yj)
Note that this is equivalent to having increasing differences—as defined in the main text—for all pairs of components. Any univariate function has increasing differences be-cause the condition requiresi =j.
We will also need to appeal to the following partial equivalence between increasing differences and supermodularity.
Lemma3. SupposeX⊂Rn is a lattice.Iff is(strictly)supermodular onX,thenf has (strictly) increasing differences on X. If X =ni=1Xi with Xi⊂R for all i and f has (strictly)increasing differences onX,thenf is(strictly)supermodular onX.
Proof. Let
×
denote the direct product (a generalization of the Cartesian product).9 Then Theorem 2.6.1 ofTopkis (1998) gives that ifXα is a lattice for eachαin a setA, X is a sublattice of×
α∈AXα, andf (x)is (strictly) supermodular onX, thenf (x)has (strictly) increasing differences onX. TakingA= {1 n}andXα=Rfor allα∈Agives×
α∈AXα=Rn(which is example 2.2.1 part (c) onTopkis(1998, p. 12)). Consequently,X is a sublattice ofRnand the theorem applies to show the (strict) supermodularity off onXimplies (strictly) increasing differences off onX.Corollary 2.6.1 ofTopkis(1998) gives that ifXiis a chain (by definition, a partially ordered set that contains no unordered pairs of elements) for i=1 n and f has (strictly) increasing differences on
×
ni=1Xi, thenfis (strictly) supermodular on×
ni=1Xi. SinceXi⊂R, it is a chain, and the direct product×
ni=1Xiis just the Cartesian productn
i=1Xi. Hence (strictly) increasing differences onX implies (strict) supermodularity
onX.
F.1.2 Proofs
Proof of Proposition 3. Because I ⊂R, it is a lattice. Additionally, −π(i i) has decreasing differences and is trivially submodular in i (as well as supermodular). So Theorem 6.1 ofTopkis (1978) gives that arg mini∈I(i)−π(i i) is ascending on the set of i such that a solution exists. Theorem 6.3 strengthens this to strongly ascending when−π(i i)has strictly decreasing differences. NotingG(i):=arg maxi∈I(i)π(i i)=
arg mini∈I(i)−π(i i)then gives the result.
Proof of Lemma1. To haveI ascending onI one needsi1< i2,i1∈I(i1), andi2∈
I(i2)to implymin{i1 i2} ∈I(i1)andmax{i1 i2} ∈I(i2). SinceIis increasing,i1∈I(i2)
and somax{i1 i2} ∈I(i2). Ifi2≥i1, then one hasmin{i1 i2} =i1∈I(i1). So, takei2< i1.
9Topkis(1998) defines the direct product, which he denotes by
×
, in this way: “IfXαis a set for eachα in a setA, then the direct product of these setsXαis the product set
×
α∈AXα= {x=(xα:α∈A):xα∈Xα for eachα∈A}” (p. 12). The notation(xα:α∈A)gives a vector that “consists of a componentxαfor eachWe need to show thati2∈I(i1)fori1< i2andi2< i1. Pick an arbitrarymand
sup-press dependence on it. Then
hi2 i2
−hi1 i2
≤hi2 i1
−hi1 i1
≤hi2 i1
≤hi2 i2
where the first line follows from increasing differences, the second fromi1∈I(i1) so
thath(i1 i1)≥0, and the third fromhbeing decreasing ini. Consequently,−h(i1 i2)≤0
which givesh(i1 i2)≥0. Since themwas arbitrary, this holds for allmand soi2∈I(i1).
Thus,min{i1 i2} ∈I(i1)and soIis ascending onI.
Proof ofLemma2. For (a) and (b), we prove (b) which implies (a). Let(i1 i1) (i2 i2)∈
Swithi1< i2andi1< i2but otherwise arbitrary. Then
fi2 ij
−fi1 ij
=p(i2)+q
ij+r(i2)s
ij−p(i1)−q
ij−r(i1)s
ij
=p(i2)−p(i1)+
r(i2)−r(i1)
sij
So,f (i2 i1)−f (i1 i1)≤f (i2 i2)−f (i1 i2)if and only if
p(i2)−p(i1)+r(i2)−r(i1)si1
≤p(i2)−p(i1)+r(i2)−r(i1)si2
⇔ 0≤r(i2)−r(i1)si2−si1
which holds becauserandsare either both increasing or both decreasing.
For (c), let(i1 i1) (i2 i2)∈Swithi1< i2andi1< i2 but otherwise arbitrary. Thenf
has increasing differences if and only if
gi2 i1
−gi1 i1
≤gi2 i2
−gi1 i2
sincegagrees withfonS. BecausegisC2, this holds if
[i1i2]g1
θ i1dθ≤
[i1i2]g1
θ i2dθ ⇔ 0≤
[i1i2]
g1
θ i2−g1
θ i1dθ
Again, bygbeingC2, this holds if
0≤
[i1i2]
[i1i2]g12
θ θdθdθ
BecauseLis assumed to be a hypercube containingSandg12≥0onL, this holds.
For (d), let(i1 i1) (i2 i2)∈Swithi1< i2andi1< i2but otherwise arbitrary. Thenf
has increasing differences if and only if
k
αkfki2 i1
−
k
αkfki1 i1
≤
k
αkfki2 i2
−
k
αkfki1 i2
⇔
k
αkfki2 i1
−fki1 i1
≤
k
αkfki2 i2
−fki1 i2
A sufficient condition for this is that, for all k, fk has increasing differences so that fk(i2 i1)−fk(i1 i1)≤fk(i2 i2)−fk(i1 i2).
For (e) and (f ), let(i1 i1) (i2 i2)∈Swithi1< i2andi1< i2but otherwise arbitrary.
The compositionh◦ghas increasing differences onSif and only if
hgi2 i1−hgi1 i1≤hgi2 i2−hgi1 i2 (22)
Becausegis increasing ini,g(i2 i)−g(i1 i)≥0. Then becausehisC2, (22) is equivalent
to
g(i2i1)−g(i1i1)
0
hgi1 i1
+θdθ≤
g(i2i2)−g(i1i2)
0
hgi1 i2
+θdθ
Becauseghas increasing differences,g(i2 i1)−g(i1 i1)≤g(i2 i2)−g(i1 i2). Hence, this
is equivalent to
g(i2i1)−g(i1i1)
0
hgi1 i1
+θ−hgi1 i2
+θdθ≤
g(i2i2)−g(i1i2)
g(i2i1)−g(i1i1)
hgi1 i2
+θdθ
Becauseh>0, the right-hand side is positive. So, a sufficient condition for this to hold is that the left-hand side be negative, which is true ifh(g(i1 i1)+θ)≤h(g(i1 i2)+θ)for all
positiveθ. In (f ),gis increasing in its second argument, sog(i1 i1)+θ≤g(i1 i2)+θ, and,
becauseh>0, this holds. In (g),gis decreasing in its second argument, sog(i1 i1)+θ≥
g(i1 i2)+θ, and becauseh<0, this holds. So,h◦ghas increasing differences.
For (g), let (i1 i1) (i2 i2)∈S withi1< i2 andi1< i2 but otherwise arbitrary. Then
becausehis increasing iniand becauseghas increasing differences,
gh(i2 ε) i1
−gh(i1 ε) i1
≤gh(i2 ε) i2
−gh(i1 ε) i2
for anyε∈E. Integrating,
E
gh(i2 ε) i1
−gh(i1 ε) i1
dF(ε)≤
E
gh(i2 ε) i2
−gh(i1 ε) i2
dF(ε)
Fromf (i i)=Eg(h(i ε) i) dF(ε), this says
fi2 i1
−fi1 i1
≤fi2 i2
−fi1 i2
which establishes thatfhas increasing differences.
Proof ofProposition4. Leta < bwitha b∈I. Define the feasible set asΓ (i):= {i∈
I|c(i i)≥0}. Letg
1∈G(a)andg2∈G(b).
To establish thatGis ascending, it is sufficient to show thatg1> g2impliesg1∈G(b)
and g2∈G(a). To establish that Gis strongly ascending, it is sufficient to show that
g1> g2gives a contradiction. So, supposeg1> g2. Unless explicitly stated, we only
as-sumecis weakly increasing ini, has weakly increasing differences, and thatW is weakly increasing.
First, we will show c(a g2)≥c(a g1)≥0and c(b g2)≥c(b g1)≥0for W weakly
increasing andc(a g2) > c(a g1)≥0andc(b g2) > c(b g1)≥0forW strictly increasing.
To see this, note thatΓ is increasing. Consequently,g1∈Γ (b)(soc(b g1)≥0), and hence
g2∈G(b)implies
uc(b g2)
+W (g2)≥u
c(b g1)
+W (g1) (23)
Becauseg2< g1andW is weakly (strictly) increasing, this impliesc(b g2)≥(>)c(b g1).
So, exploiting weakly increasing differences, 0≥ (>)c(b g1) −c(b g2) ≥ c(a g1) −
c(a g2)forW weakly (strictly) increasing. Also usingg1∈Γ (a),c(a g2)≥(>)c(a g1)≥0
forW weakly (strictly) increasing. For use below, note also that becauseg1∈G(a)and
g2∈Γ (a),
uc(a g1)
+W (g1)≥u
c(a g2)
+W (g2) (24)
Combining (23) and (24),
uc(b g2)
+W (g2)−u
c(b g1)
−W (g1)≥0≥u
c(a g2)
+W (g2)−u
c(a g1)
−W (g1)
which implies
uc(b g2)−uc(b g1)≥uc(a g2)−uc(a g1) (25)
As established above,c(b g2)≥c(b g1)andc(a g2)≥c(a g1). Using this and the
differ-entiability ofu, (25) is equivalent to
c(bg2)−c(bg1)
0
uc(b g1)+θdθ≥
c(ag2)−c(ag1)
0
uc(a g1)+θdθ
Because of weakly increasing differences, c(a g1)−c(a g2)≤ c(b g1)−c(b g2) or,
equivalently,c(b g2)−c(b g1)≤c(a g2)−c(a g1). Moreover,c(b g2)≥c(b g1). So, the
above inequality is equivalent to
0≥
c(ag2)−c(ag1)
c(bg2)−c(bg1)
uc(a g1)+θdθ
+
c(bg2)−c(bg1)
0
uc(a g1)+θ−uc(b g1)+θdθ
Becausecis weakly increasing inianduis concave, the second integral is positive. The first must also be positive. Hence, the inequality holds if and only if
c(a g2)−c(a g1)=c(b g2)−c(b g1) (C1)
Now, consider the claims again. For the second claim, we seek a contradiction. A contradiction obtains ifchas strictly increasing differences as then (C1) is violated. Alternatively, ifW is strictly increasing andcis strictly increasing ini, (C2) will be vio-lated becausec(b g2) > c(b g1)andc(a g1) > c(b g1).
For the first claim, we want to show g1 ∈G(b) and g2∈ G(a). (C2) implies
ei-ther c(b g2)=c(b g1) and/or c(a g1)=c(b g1). Consider the cases separately with
c(b g2) =c(b g1) first. Then (C1) gives c(a g2) −c(a g1) =c(b g2) −c(b g1). So,
c(a g2)=c(a g1). Hence, the choices give the same consumption ata andb. So, the
continuation utility must be the same: equation (23) implies W (g2)≥W (g1)and (24)
impliesW (g1)≥W (g2). So, with the same consumption and choice utilities,g1∈G(a)
givesg2∈G(a)andg2∈G(b)givesg1∈G(b).
Now consider the second case wherec(a g1)=c(b g1). Because (C1) givesc(a g2)−
c(a g1)=c(b g2)−c(b g1), replacingc(a g1) with c(b g1) gives c(a g2)−c(b g1)=
c(b g2)−c(b g1)orc(a g2)=c(b g2). Then
uc(a g1)+W (g1)≥uc(a g2)+W (g2)
⇔ uc(b g1)
+W (g1)≥u
c(b g2)
+W (g2)
(26)
where the first line follows from the optimality of g1 ∈ G(a) and the second from
c(a g1)=c(b g1)andc(a g2)=c(b g2). Consequently, sinceg2∈G(b)and (26) shows
g1delivers weakly higher utility atb,g1∈G(b).
To establish g2 ∈G(a), the argument is similar. We have c(a g1)=c(b g1) and
c(a g2)=c(b g2). Because we have showng1∈G(b),u(c(b g1))+W (g1)=u(c(b g2))+
W (g2). Replacingc(b g1)withc(a g1)andc(b g2)withc(a g2), this becomes
uc(a g1)
+W (g1)=u
c(a g2)
+W (g2)
Consequently,g1∈G(a)impliesg2∈G(a).
Proof of Proposition 5. Becausef is assumed to be differentiable (left and right) on the closure of S, Theorem 1 of Milgrom and Segal (2002) gives that fi(i i) = u1(g(x∗(i i) i i))g2(x∗(i i) i i)onS.
To show increasing differences, we need to establish that
fi2 i1
−fi1 i1
≤fi2 i2
−fi1 i2
for(i1 i1) (i2 i2)∈Swithi1≤i2andi1≤i2. SincefisC1ini, this is equivalent to
0≤ i2
i1
fiθ i2−fiθ i1dθ
Hence, iffiis increasing ini, then increasing difference holds. Definingx∗j :=x∗(i ij), thenfiis increasing iniif
0≤u1
gx∗2 i i2g2
x∗2 i i2−u1
gx∗1 i i1g2
x∗1 i i1
=u1gx∗2 i i2
g2x∗2 i i2
−g2x∗1 i i1
+u1gx∗2 i i2
−u1gx∗1 i i1
g2x∗1 i i1
Sinceu1≥0andg2(x∗(i i) i i)is increasing ini, the first term is positive. Sinceu11≤0,
g2(x∗(i i) i i)≥0, andg(x∗(i i) i i)is decreasing ini, the second term is positive. So,
f has increasing differences.
F.2 Algorithm correctness proofs and lemmas
We now give the omitted proofs and lemmas pertaining to the algorithm correctness. Let the objective function π(i i˜ ), the maximumΠ(i), and the feasible choice set˜ I(i)be as in (7). Assume thatI(i)⊂ {1 n}—which may be empty—is monotonically increasing, and defineI := {i∈ {1 n}|I(i) = ∅}so thati∈I has a feasible solution. For alli∈I, defineG(i)˜ :=arg maxi∈I(i)π(i i˜ ). Letπbe defined via (8)—whereπis such
thatπ(i i˜ ) > πfor alli∈I andi∈I(i)—withΠ(i):=maxi∈{1 n}π(i i)andG(i):=
arg maxi∈{1 n}π(i i). Note that by construction,i=1is optimal whenever there is no
feasible choice andi∈I(i)is always preferable toi∈/I(i)when a feasible choice exists. Lemma4establishes the mathematical equivalence of these problems (but does not say that the algorithms applied to (1) deliver a correct solution to (7)).
Lemma4. All of the following are true:
1. Π(i)= ˜Π(i)for alli∈I.
2. G(i)= ˜G(i)for alli∈I.
3. IfG˜ is ascending onI,thenGis ascending on{1 n}.
Proof. The first claim is an implication of the second claim. For the proof of the second claim, leti∈I. ThenI(i) = ∅. Infeasible choices, that is,i∈ {1 n} \I(i), are strictly suboptimal in theΠproblem (1) because any feasible choicej∈I(i)deliversπ(i j) > π=π(i i). Hence,
G(i)= arg max
i∈{1 n}
πi i=arg max
i∈I(i)
πi i=arg max
i∈I(i) ˜
πi i= ˜G(i)
(where the third equality follows from the definition ofπ).
To show the third claim, letG˜ be ascending onI. Now, leti1≤i2andg1∈G(i1)and
g2∈G(i2). We want to show thatmin{g1 g2} ∈G(i1) and max{g1 g2} ∈G(i2). Clearly,
this is the case if g1≤g2, so takeg1> g2. Then since G(i)= {1}for alli /∈I andg1>
g2≥1, it must be thati1∈I (otherwise,g1would have to be1). Then, because I(i)is
increasing, i2∈I. Hence,G(i1)= ˜G(i1) andG(i2)= ˜G(i2). So,G˜ ascending gives the
desired result.
Lemmas5and6establish that, for concave problems, simple and binary concavity deliver an optimal choice provided there is one in the search space.
Lemma5. If the problem is concave andπ(i j)≥π(i j+1)for somej,then π(i j)=
maxi∈{j n}π(i i) (j is as least as good as anything to the right of it).Ifπ(i k−1) < π(i k)for somek,thenπ(i k)=maxi∈{1 k}π(i i)=maxi∈{1 k}π(i i˜ )(kis as least as
Proof. Note that by the problem being concave (as given in Definition3), there is some increasing functionn(i)such thatI(i)= {1 n(i)}fori∈I.
To proveπ(i j)≥π(i j+1)impliesπ(i j)=maxi∈{j n}π(i i), consider two cases.
First, suppose i /∈I. Then I(i)= ∅ and π(i i)=1[i =1]. Consequently, for any j, π(i j)=maxi∈{j n}π(i i). In other words,jis weakly better than any value to the right
of it.
Second, supposei∈I. Ifj > n(i), thenπ(i j)=π=maxi∈{j n}π(i i). Ifj=n(i),
then π(i j) > π =maxi∈{j+1 n}π(i i) implying π(i j)=maxi∈{j n}π(i i). If j < n(i), then
max
i∈{j n}π
i i=max max
i∈{j n(i)}π
i i max
i∈{n(i)+1 n}π
i i
=max max
i∈{j n(i)}π
i i π
= max
i∈{j n(i)}π
i i
= max
i∈{j n(i)}π˜
i i
All that remains to be shown for this case isπ(i j)=maxi∈{j n(i)}π(i i˜ ). Sinceπ(i j)=
˜
π(i j)andπ(i j+1)= ˜π(i j+1), the hypothesis givesπ(i j)˜ ≥ ˜π(i j+1). Because the problem is concave andπ(i˜ ·)is weakly decreasing fromjtoj+1, it must be weakly decreasing fromjton(i). Hence, π(i j)˜ =maxi∈{j n(i)}π(i i˜ ). So,π(i j)= ˜π(i j)= maxi∈{j n(i)}π(i i˜ )=maxi∈{j n}π(i i).
Now, we prove the case forπ(i k−1) < π(i k). In this case,k−1andkmust both be feasible, that is,k≤n(i), because (1) if they were both infeasible, thenπ(i k−1)= π=π(i k)and (2) if onlykwere infeasible, thenπ(i k−1) > π=π(i k). Given that k−1and kare feasible, π(i k−1)= ˜π(i k−1) andπ(i k)= ˜π(i k). Sinceπ(i˜ ·) is strictly increasing until it switches to weakly decreasing, π(i˜ 1) <· · ·<π(i k˜ −1) <
˜
π(i k). Henceπ(i k)˜ =maxi∈{1 k}π(i i˜ ). Since all of1 kare feasible,π(i k)=
maxi∈{1 k}π(i i)=maxi∈{1 k}π(i i˜ ).
Lemma6. Suppose it is known thatG(i)∩ {a b}is nonempty.Then brute force ap-plied to
max
i∈{a b}π
i i
delivers an optimal solution,that is,lettinggˆ be the choice the algorithm delivers,gˆ∈ G(i).Additionally,if the problem is concave,then the simple concavity and binary con-cavity algorithms also deliver an optimal solution.
If so, thenb−a+1=3and it does a comparison of either (1)aandm=(b+a)/2, in which case it correctly identifies the maximum asaor (2)mandbin which case it drops bfrom the search space and does a brute force comparison ofaanda+1(when it goes to step 2). Ifb−a+1>3, it will evaluate the midpointm= (a+b)/2andm+1and findπ(i m)=π(i m+1)=0. It will then proceed to step 2, searching for the optimum in{1 m}witha=1andb=min the next iteration of the recursive algorithm. This proceeds untilb−a+1≤3, where it then correctly identifies the maximum (as was just discussed). Therefore, binary concavity finds a correct solution,gˆ∈G(i).
Now, supposei∈I. Because brute force will evaluateπ(i·)at every i∈ {a b}, it finds gˆ ∈G(i). Now, suppose the problem is concave. The simple concavity algo-rithm evaluatesπ(i i)ati∈ {a b}sequentially until it reaches ax∈ {a+1 b} that π(i x−1)≥π(i x). If this stopping rule is not triggered, then simple concavity is identical to brute force and so finds an optimal solution. So, it suffices to consider otherwise. In this case, x−1satisfies the conditions for “j” in Lemma5, and hence π(i x−1)=maxi∈{x−1 n}π(i i). By virtue of not having stopped untilx−1,π(i x− 1)≥maxi∈{a x−1}π(i i). Consequently,π(i x−1)≥maxi∈{a x−1}∪{x−1 n}π(i i)=
maxi∈{a n}π(i i). Since a maximum is known to be in{a b},
Π(i)= max
i∈{a b}π
i i≤ max
i∈{a n}π
i i≤π(i x−1)≤Π(i)
So,π(i x−1)=Π(i)givingx−1∈G(i).
Now consider the binary concavity algorithm. Ifb≤a+1(so that the size of the search space,b−a+1, is1or2), the algorithm is the same as brute force and so finds a maximum. Ifb=a+2(a search space of size3), the algorithm goes to either step 3(a) or step 3(b). In step 3(a), it stops ifπ(i a) > π(i m) (wherem=(a+b)/2) taking the maximum asa and otherwise does the same as brute force. So, suppose the stopping condition is satisfied. A maximum isaas long asπ(i a)=maxi∈{a b}π(i i), which it is
sinceasatisfies the conditions for “j” in Lemma5. In step 3(b), it stops ifπ(i b) > π(i m) taking the maximum asband otherwise does the same as brute force. So, suppose the stopping condition is satisfied. A maximum isbas long asπ(i b)=maxi∈{a b}π(i i), which is true sincebsatisfies all the conditions for “k” in Lemma5.
We now give the proof of Proposition6, which establishes that the monotonicity and concavity algorithms deliver an optimal policy.
Proof ofProposition6. We will show, lettinggˆbe the policy function the algorithm finds, thatg(i)ˆ ∈G(i)for alli, which impliesg(i)ˆ ∈ ˜G(i)for alli∈I by Lemma4. Each of the brute force, simple, and binary monotonicity algorithms can be thought of as iterating through statesi(in some order that, in the case of binary monotonicity, de-pends onπ) with a search space{a b}. If every state is visited and optimal choice is found at each state, then an optimal solution is found. So, it suffices to show that each of the brute force, simple, and binary monotonicity algorithms explore every state i∈ {1 n}and at each state, the following conditions are met so that Lemma6can be applied: (1){a b} ⊂ {1 n}; (2)a≤b; and (3)G(i)∩ {a b} = ∅. An applica-tion of Lemma6then givesg(i)ˆ ∈G(i)(provided an appropriate concavity algorithm is used).
Brute force monotonicity trivially explores all states i∈ {1 n} sequentially. At eachi,a=1andb=n. Consequently,G(i)∩ {a b} = ∅and Lemma6can be applied. Now, we prove simple monotonicity and binary monotonicity deliver a correct solu-tion whenG˜ is ascending, which, by Lemma4, gives thatGis ascending.
The simple monotonicity algorithm explores all statesi∈ {1 n}sequentially al-ways with b=n (and soa≤b). Fori=1,a=1and soG(1)∩ {a b} = ∅. Conse-quently, Lemma6gives thatg(ˆ 1)∈G(1). Now, consider somei >1and suppose for in-duction thatg(iˆ −1)∈G(i−1). Because Gis ascending andg(iˆ −1)∈G(i−1), any
˙
g∈G(i)impliesmax{ ˆg(i−1)g} ∈˙ G(i). So,G(i)∩ {g(i−1) n} = ∅. Hence, Lemma6
applies, andg(i)ˆ ∈G(i)completing the induction argument.
Now consider the binary monotonicity algorithm. Ifn=1orn=2, the algorithm is the same as simple monotonicity and so delivers a correct solution. Ifn >2, then the al-gorithm first correctly identifiesg(ˆ 1)(by brute force) andg(n)ˆ (using the same argument as simple monotonicity). It then definesi=1andi=nand maintains the assumption thatg(i)ˆ ∈G(i)andg(i)ˆ ∈G(i).
The goal of step 2 is to find the optimal solution for all i∈ {i i}. The algo-rithm stops at step 2(a) if i≤i+1, in which case this objective is clearly met since {i i} = {i i}. If the algorithm does not stop, then it computesg(m)ˆ form= (a+b)/2
using the search space { ˆg(i) g(i)ˆ }. By Lemma6, an optimum is found as long as G(m)∩ { ˆg(i) g(i)ˆ } = ∅. Ifg(i)ˆ ∈G(i)andg(i)ˆ ∈G(i), then for anyg˙∈G(m),G as-cending givesmin{ ˆg(i)max{ ˆg(i)g˙}} ∈G(m). So,G(m)∩ { ˆg(i) g(i)ˆ } = ∅ifg(i)ˆ ∈G(i) and g(i)ˆ ∈G(i). This holds because of the algorithm’s maintained assumptions.10 So, if everyi∈ {2 n−1}is the midpoint of some(i i)after iterating some number of times, the proof is complete. In other words, since the algorithm only solves for the op-timal policy at midpoints once it reaches step 2, we need to prove every state (except for i=1andi=n) is eventually a midpoint.
10Formally, it can be shown through induction. It is true at the first instance of step 2. Since in step 2(c)
the algorithm then divides into{i m}and{m i}, it is also true at the next iteration. Consequently,
To show that everyi∈ {2 n−1}is a midpoint of some interval reached in the recursion, fix an arbitrary suchiand suppose not. Define(i1 i1)=(1 n). When step 2 is
first reached,i∈ {i1+1 i1−1}. Now, uniquely and recursively define(ik ik)to be the one of(ik−1 m)and(m ik−1)withm= (ik−1+ik−1)/2such thati∈ {ik+1 ik−1} (becauseiis assumed to never be a midpoint, this is well defined).
Now, consider the cardinality of{ik ik}defining it asNk=ik−ik+1. By con-struction,i∈ {ik+1 ik−1}for eachk. So, a contradiction is reached if{ik+1 ik−
1} = ∅ which is equivalent to Nk ≤2. So, it must be that Nk ≥ 3 for all k. If Nk−1
is odd, then Nk=(Nk−1+1)/2. If Nk−1 is even, Nk≤Nk−1/2+1. So, in either case
Nk≤Nk−1/2+1. DefiningMk recursively byM1=N1 andMk=Mk−1/2+1, one can
show by induction thatNk≤Mk for allk. BecauseNk≥3for allk, Mk≥3for allk. HenceMk−Mk−1=1−Mk−1/2≤1−3/2= −1/2. HenceMk≤Mk−1−1/2. Therefore,
Mkwill be less than three in a finite number of iterations, which gives a contradiction.
F.3 Cost bound proofs and lemmas
We now give the cost bound proofs and supporting lemmas. SectionF.3.1establishes the performance ofHeer and Maußner’s (2005) binary concavity. SectionF.3.2proves the performance of the one-state binary monotonicity as stated in Proposition 1. Sec-tion F.3.3 provesthe performance of the two-state binary monotonicity as stated in Proposition 2.
F.3.1 Binary concavity
Lemma7. Consider the problemmaxi∈{a a+n−1}π(i i)for anyaand anyi.For anyn∈ Z++,binary concavity requires no more2log
2(n) −1evaluations ifn≥3and no more
thannevaluations ifn≤2.
Proof. Forn=1, the algorithm computes π(i a)and stops, so one evaluation is re-quired. Forn=2, two evaluations are required (π(i a)andπ(i a+1)). Forn=3, step 3 requiresπ(i m)to be computed and may requireπ(i a)to be computed. Then step 3(a) or step 3(b) either stop with no additional function evaluations or go to step 2 with
max{1a1b} =1where, in that case, at most one additional function evaluation is re-quired. Consequently,n=3requires at most three function evaluations, which agrees with2log2(3) −1=3. So, the statement of lemma holds for1≤n≤3.
evaluations, or step 4 (withn=4), requiring at most three evaluations. So, the evalua-tions are weakly less than5=2+max{23}. Hence, for everyn=456, and7, the re-quired evaluations are less than3,4,4, and5, respectively. One can then verify that the evaluations are less than2log2(n) −1for these values ofn.
Now, supposen≥4. We shall prove that the required number of evaluations is less thanσ(n):=2log2(n) −1by induction. We have already verified the hypothesis holds for n∈ {4567}, so consider some n≥ 8 and suppose the hypothesis holds for all m∈ {4 n−1}. Letibe such thatn∈ [2i+12i+1]. Then note that two things are true,
log2(n) =i+1andlog2(n+21) =i.11Sincen≥4, the algorithm is in (or proceeds to) step 4, which requires two evaluations, and then proceeds with a new interval to step 4 (again). Ifnis even, the new interval has sizen/2. Ifnis odd, the new interval either has a size of(n+1)/2or(n−1)/2. So, ifnis even, no more than2+σ(n/2)evaluations are required; ifnis odd, no more than2+max{σ((n+1)/2) σ((n−1)/2)} =2+σ((n+1)/2) evaluations are required. The even and odd case can then be handled simultaneously with the bound2+σ(n+21). Manipulating this expression using the previous
observa-tion thatlog2(n) =i+1andlog2(n+21) =i,
2+σ
n+1 2
=2+2
log2
n+1 2
−1
=2+2i−1
=2log2(n)−1
Hence, the proof by induction is complete.
F.3.2 Binary monotonicity in one dimension In proving the performance on the one-state binary monotonicity algorithm, we allow for many maximization techniques by characterizing the algorithm’s properties conditional on a monotonically increasingσ:
Z++→Z+that bounds the evaluation count required to solve (3).
Because of the recursive nature of binary monotonicity, theπevaluation bound for generalσis also naturally recursive. In Proposition7, we will show the algorithm’s cost is2σ(n)+Mσ(n n)where the functionMσ is defined as follows.
Definition 5. For anyσ:Z++→Z+, defineMσ : {23 } ×Z+→Z+recursively by Mσ(z γ)=0ifz=2orγ=0and
Mσ(z γ)=σ(γ)+ max
γ∈{1 γ}
Mσ
z
2
+1 γ
+Mσ
z
2
+1 γ−γ+1
forz >2andγ >0.
Now, we establish thatMσis increasing.
11The proof is as follows. Both log
2(·) and log2(·) are weakly increasing functions. So, n∈
[2i +12i+1] implies log
2(n) ∈ [log2(2i+1)log2(2i+1)] = [i+1 i+1]. Likewise, log2(n+21) ∈
[log2(2 i+1+1
2 )log2(2
i+1+1
Lemma8. For anyσ,Mσ(z γ)is weakly increasing inzandγ.
Proof. Fix aσ and suppress dependence on it. First, we will showM(z γ)is weakly increasing inγfor everyz. The proof is by induction. Forz=2,M(2·)=0. Forz=3, M(3 γ)=σ(γ)which is weakly increasing inγ. Now consider somez >3and suppose M(y·)is weakly increasing for ally≤z−1. Forγ2> γ1,
M(z γ1)=σ(γ1)+ max
γ∈{1 γ1}
M z 2
+1 γ
+M z 2
+1 γ1−γ+1
≤σ(γ2)+ max
γ∈{1 γ2}
M z 2
+1 γ
+M z 2
+1 γ2−γ+1
=M(z γ2)
where the inequality is justified byσbeing increasing and the induction hypothesis giv-ingM(z2 +1·)as an increasing function (notez2 +1≤z−1for allz >3).
Now we will showM(z γ)is increasing inzfor everyγ. The proof is by induction. First, note thatM(2 γ)=0≤σ(γ)=M(3 γ)for allγ >0andM(2 γ)=0=M(3 γ)for γ=0. Now, consider somek >3and suppose that for anyz1 z2≤k−1with z1≤z2
thatM(z1 γ)≤M(z2 γ)for allγ. The goal is to show that for anyz1 z2≤kwithz1≤z2
thatM(z1 γ)≤M(z2 γ)for allγ. So, consider suchz1 z2≤kwithz1≤z2. Ifγ=0, then
M(z1 γ)=0=M(z2 γ), so takeγ >0. Then
M(z1 γ)=σ(γ)+ max
γ∈{1 γ} M z1 2
+1 γ
+M z1 2
+1 γ−γ+1
≤σ(γ)+ max
γ∈{1 γ} M z2 2
+1 γ
+M z2 2
+1 γ−γ+1
=M(z2 γ)
The inequality obtains sincezi2 +1≤k−1for alli(which is true since even ifzi=k, one hask/2 +1≤k−1by virtue ofk >3). So, the induction hypothesis givesM(z1
2 +
1·)≤M(z2
2 +1·), and the proof by induction is complete.
Now we can give a cost bound for the one-state binary monotonicity algorithm.
Proposition7. Letσ:Z++→Z+be an upper bound on the number ofπevaluations required to solve(3).Then the algorithm requires at most2σ(n)+Mσ(n n)evaluations forn≥2andn≥1.
Proof. Sincegis the policy function associated with (1),g: {1 n} → {1 n}. By monotonicity,gis weakly increasing. DefineN: {1 n}2→Z+by
N(a b)=Mb−a+1 g(b)−g(a)+1
noting that this is well defined (based on the definition ofM) wheneverb > a. Addition-ally, define a sequence of setsIkfork=1 n−1by
Ik:=
Note that for anyk∈ {1 n−1},Ik is nonempty andN(a b)is well defined for any (a b)∈Ik.
We shall now prove that for anyk∈ {1 n−1},(i i)∈IkimpliesN(i i)is an upper bound on the number of evaluations ofπrequired by the algorithm in order to compute the optimal policy for alli∈ {i i}wheng(i)andg(i)are known. If true, then begin-ning at step 2 in the algorithm (which assumesg(i)andg(i)are known) with(i i)∈Ik, N(i i)is an upper bound on the number ofπevaluations.
The argument is by induction. First, considerk=1. For any(a b)∈I1, the algorithm
terminates at step 2(a). Consequently, the number of required π evaluations is zero, which is the same asN(a b)=M(b−a+1 g(b)−g(a)+1)=M(2 g(b)−g(a)+1)=0
(recallM(2·)=0).
Now, consider somek∈ {2 n−1}and suppose the induction hypothesis holds for all jin 1 k−1. That is, assume for allj in1 k−1that (i i)∈Ij implies N(i i)is an upper bound on the number of requiredπevaluations wheng(i)andg(i) are known. We shall show it holds forkas well.
Consider any (i i)∈Ik with g(i) and g(i) are known. Since i > i+1, the algo-rithm does not terminate at step 2(a). In step 2(b), to computeg(m)(wherem:= i+2i), one must find the maximum within the range g(i) g(i), which requires at most σ(g(i)−g(i)+1)evaluations ofπ. In step 2(c), the space is then divided into{i m} and{m i}.
If k is even, thenm equals i+2i. Since (i m)∈Ik/2 and g(i) and g(m)are known,
the induction hypothesis givesN(i m)as an upper bound on the number ofπ eval-uations needed to computeg(i) g(m). Similarly, since(m i)∈Ik/2 andg(m)and
g(i) are known, N(m i) provides an upper bound on the number of π evaluations needed to computeg(m) g(i). Therefore, to computeg(i) g(i), at mostσ(g(i)− g(i)+1)+N(i m)+N(m i)evaluations are required. Definingγ=g(i)−g(i)+1and γ=g(m)−g(i)+1and using the definition ofmandN, we have that the number of required evaluations is less than
σ(γ)+Mm−i+1 g(m)−g(i)+1+Mi−m+1 g(i)−g(m)+1
=σ(γ)+M
i+i
2 −i+1 γ
+Mi−i+i
2 +1 g(i)−g(m)+1
=σ(γ)+M
i−i
2 +1 γ
+Mi−i
2 +1 g(i)−g(m)+γ
−γ+1
=σ(γ)+M
i−i
2 +1 γ
+Mi−i
2 +1 g(i)−g(m)+g(m)−g(i)+1−γ +1
=σ(γ)+M
i−i
2 +1 γ
+Mi−i
2 +1 γ−γ
+1
evalua-tions is not greater than
σ(γ)+ max
γ∈{1 γ}
M
i−i
2 +1 γ
+Mi−i
2 +1 γ−γ
+1
The case ofk odd is very similar, but the divide-and-conquer algorithm splits the space unequally. Ifkis odd, thenmequalsi+i2−1. In this case(i m)∈I(k−1)/2and(m i)∈ I(k−1)/2+1.12Consequently, computing the policy fori itakes no more thanσ(g(i)−
g(i)−1)+N(i m)+N(m i)maximization steps. Definingγandγthe same as before and using the definition ofmandN, we have that the required maximization steps is less than
σ(γ)+Mm−i+1 γ+Mi−m+1 γ−γ+1
=σ(γ)+M
i+i−1
2 −i+1 γ
+Mi−i+i−1
2 +1 γ−γ
+1
=σ(γ)+M
i−i+1
2 γ
+Mi−i+1
2 +1 γ−γ
+1
BecauseMis increasing in the first argument, this is less than
σ(γ)+ max
γ∈{1 γ}
M
i−i+1
2 +1 γ
+Mi−i+1
2 +1 γ−γ
+1
Combining the bounds forkeven and odd, the required number ofπevaluations is less than
σ(γ)+ max
γ∈{1 γ}
M
i−i+1 2
+1 γ
+M
i−i+1 2
+1 γ−γ+1
(27)
because ifkis even, theni−2i+1 =i−2i. Consequently, (27) gives an upper bound for any (i i)∈Ikfork≥1wheng(i)andg(i)are known. IfN(i i)is less than this, then the proof by induction is complete.
SinceN(i i)is defined asM(i−i+1 g(i)−g(i)+1), using the definitions ofNand Mshows
N(i i)=Mi−i+1 g(i)−g(i)+1
=M(i−i+1 γ)
=σ(γ)+ max
γ∈{1 γ}
M
i−i+1 2
+1 γ
+M
i−i+1 2
+1 γ−γ+1
Consequently,N(i i)exactly equals the value in (27), and the proof by induction is com-plete.
12To see this, note that(i i)∈I
kimpliesk=i−i. To have,(i m)∈I(k−1)/2, it must be thatm=i+k−21.
This holds:i+k−21=i+i−i2−1=i+i+i−21−i−i=i+m+−22i=m. Similarly, to have(m i)∈I(k−1)/2+1, one
Step 1 of the algorithm requires at most 2σ(n) evaluations to computeg(1) and g(n). Ifn=2, step 2 is never reached. SinceM(n n)=0in this case,2σ(n)+M(n n) provides an upper bound. Ifn >2, then since(1 n)∈In−1andg(1)andg(n) known,
only N(1 n) additional evaluations are required. Therefore, to compute for each i∈ {1 n}, no more than2σ(n)+N(1 n)=2σ(n)+M(n g(n)−g(1)+1)function eval-uations are needed. Lemma (8) then gives that this is less than2σ(n)+M(n n)since
g(n)−g(1)+1≤n−1+1=n.
The preceding proposition gives a fairly tight bound. However, it is unwieldy because of the discrete nature of the problem. By bounding σ whose domain isZ++ with aσ¯ whose domain is[1∞), a more convenient bound can be found. We give this bound in Lemma11. For its proof, we need the following two lemmas.
Lemma9. Define a sequence{mi}∞i=1bym1=2andmi=2mi−1−1fori≥2.Thenmi=
2i−1+1andlog2(mi−1)=i−1for alli≥1.
Proof. The proof ofmi=2i−1+1for alli≥1is by induction. Fori=1,m1is defined
as2, which equals21−1+1. Fori >1, suppose it holds fori−1. Thenmi=2mi−1−1=
2[2i−2+1] −1=2i−1+1.
Lemma 10. Consider anyz≥2.Then there exists a unique sequence {ni}Ii=1 such that n1=2,nI=z,n2i +1=ni−1,andni>2for alli >1.Moreover,I= log2(z−1) +1.
Proof. The proof that a unique sequence exists is by construction. Letz≥2be fixed. Define an infinite sequence{zi}∞i=1recursively as follows: definezi=Ti(z)for alli≥1
withT1(z):=zandTi+1(z)= Ti2(z) +1. We now establish all of the following:Ti(z)≥2,
Ti(z)≥Ti+1(z), andTi(z) > Ti+1(z)wheneverTi(z) >2. As an immediate consequence,
for any z≥2, there exists a uniqueI(z)≥1such that TI(z) =2and, for all i < I(z), Ti(z) >2. We also show for later use thatTi(z)is weakly increasing inzfor everyi.
To showTi(z)≥2, the proof is by induction. We haveT1(z)=zandz≥2. Now,
con-sider somei >1and suppose it holds fori−1. ThenTi(z)= Ti−1(z)
2 +1≥
2
2 +1=2.
To showTi(z) > Ti+1(z)wheneverTi(z) >2, consider two cases. First, considerTi(z)
even. ThenTi+1(z)= Ti2(z) +1=Ti2(z) +1and soTi+1(z) < Ti(z)as long asTi(z) >2.
Second, considerTi(z)odd. ThenTi+1(z)= Ti(z)2 +1=Ti(z)2−1+1and soTi+1(z) < Ti(z)
as long asTi(z) >1.
To show that Ti(z)≥Ti+1(z), all we need to show now is that Ti+1(z)=2 when
Ti(z)=2(since the inequality is strict if Ti(z) >2andTi(z)≥2for alli). If Ti(z)=2, thenTi+1(z)= 22 +1=2.
To establish thatTi(z)is weakly increasing inzfor everyi, the proof is by induction. Fora≤b,T1(a)=a≤b=T1(b). Now consider somei >1and suppose the induction
hypothesis holds fori−1. ThenTi(a)= Ti−1(a)/2 +1≤ Ti−1(b)/2 +1=Ti(b).
The sequence {nj}I(z)j=1 defined by nj =TI(z)−j+1(z)—that is, an inverted version
TI(z)−(i−1)+1= TI(z)2−(i−1) +1= n2i +1. Also, by the definition ofI(z),Ti(z) >2for any
i > I(z). So, if we can show thatI(z)= log2(z−1) +1, the proof is complete.
The proof ofI(z)= log2(z−1)+1is as follows. Note that forz=2, the sequence{zi} is simplyzi=2for alliwhich impliesI(2)=1. Sincelog2(2−1)+1=1, the relationship holds forz=2. So, now considerz >2. The proof proceeds in the following steps. First, for the special{mi}sequence defined in Lemma9, we showTj(mi)=mi+1−j for any i≥1and anyj≤i. Second, we use this to show thatI(mi)=ifor alli≥1. Third, we show thatz∈(mi−1 mi]impliesI(z)=iby showingI(mi−1) < I(z)≤I(mi). Fourth,
we show that theisuch thatz∈(mi−1 mi]is given bylog2(z−1) +1. This then gives
I(z)= log2(z−1) +1sinceI(z)=I(mi)=i= log2(z−1) +1.
First, we showTj(mi)=mi+1−jfor anyi≥1and anyj≤i. Fix somei≥1. The proof is by induction. Forj=1,T1(mi)=mi=mi+1−1. Now consider somejhaving2≤j≤i
and suppose the induction hypothesis holds forj−1. Then
Tj(mi)=
T
j−1(mi)
2
+1
=
mi
+1−(j−1)
2
+1
=
mi +2−j
2
+1
=
2mi
+1−j−1
2
+1
=mi+1−j
which provesTj(mi)=mi+1−j forj≤i. The fourth equality follows from the definition of{mi}in Lemma9.
Second, we showI(mi)=ifor alli≥1. Fix anyi≥1. We just showedTj(mi)=mi+1−j. Hence,Ti(mi)=m1=2andTi−1(mi)=m2=3. Consequently, the definition ofI—which
for a givenzis defined as the smallesti≥1such thatTi(z)=2—givesI(mi)=i(recallTj is decreasing inj).
Third, we show that z∈(mi−1 mi]implies I(z)=iby showingI(mi−1) < I(z)≤
I(mi). Note that, sincez >2(having taken care of thez=2case already), there is some i≥2such that z∈ (mi−1 mi] (sincem1=2). To see I(z)≤I(mi), suppose not, that
I(z) > I(mi). But then2=TI(z)(z) < TI(mi)(z)≤TI(mi)(mi)=2, which is a contradiction. Therefore,I(z)≤I(mi).
To seeI(mi−1) < I(z), we begin by showingTj(mi−1) < Tj(mi−1+ε)for anyε >0and
anyj≤i−1. SinceTj(mi−1)=mi−j, it is equivalent to show thatmi−j< Tj(mi−1+ε),
which we show by induction. Clearly, forj=1, we havemi−1< mi−1+ε=T1(mi−1+ε).
Now considerj >1and suppose it is true forj−1. Then
Tj(mi−1+ε)=
Tj
−1(mi−1+ε) 2
+1
=
T
j−1(mi−1+ε)−mi−j+1+mi−j+1 2
=
Tj
−1(mi−1+ε)−mi−j+1+2mi−j−1
2
+1
=
Tj
−1(mi−1+ε)−mi−j+1−1 2
+mi−j+1
Now, since the induction hypothesis ofTj−1(mi−1+ε) > mi−j+1givesTj−1(mi−1+ε)−
mi−j+1−1≥0, one has
Tj(mi−1+ε)≥
0 2
+mi−j+1=mi−j+1> mi−j
Hence the proof by induction is complete.
Now, having established Tj(mi−1) < Tj(mi−1+ε) for any ε >0and any j≤i−1,
we showI(mi−1) < I(z). Suppose not, thatI(mi−1)≥I(z). Then sincez > mi−1, taking
ε=z−mi−1we have2=TI(mi−1)(mi−1) < TI(mi−1)(mi−1+ε)=TI(mi−1)(z)≤TI(z)(z)=2,
which is a contradiction.
Lastly, we now show that theisuch thatz∈(mi−1 mi]is given bylog2(z−1) +1.
That this holds can be seen as follows. Note thatz∈(mi−1 mi]implies log2(z−1)+ 1∈(log2(mi−1−1)+1log2(mi−1)+1]. Then, sincelog2(mj−1)+1=jfor all j≥1 (Lemma9), we have log2(z−1)+1∈(i−1 i]. Then, by the definition of·, one has log2(z−1)+1 =i, which of course is equivalent tolog2(z−1) +1=i.
We established theisuch thatz∈(mi−1 mi]isi= log2(z−1) +1. Also we showed
i−1=I(mi−1) < I(z)≤I(mi)=i. HenceI(z)= log2(z−1) +1, which completes the
proof.
Now we give a more convenient bound than the one in Proposition7. The main re-sult will apply it withσ¯ bounds corresponding to brute force and binary concavity.
Lemma11. Supposeσ¯ : [1∞)→R+is either the identity map(σ(γ)¯ =γ)or is a strictly increasing,strictly concave,and differentiable function.Ifσ(γ)¯ ≥σ(γ)for allγ∈Z++, then an upper bound on function evaluations is
3σ¯n+ I−2
j=1
2jσ¯2−jn−1+1
ifI >2whereI= log2(n−1) +1.An upper bound forI≤2is3σ(n¯ ).
Proof. In keeping with the notation of the other proofs, letz andγ correspond ton andn, respectively. Fix some arbitraryz≥2. By Lemma10, there is a strictly monotone increasing sequence{zi}Ii=1withzI=z,zi= zi+1
2 +1fori < I, and withI= log2(z− 1) +1(and havingz1=2).
Fori >1and anyγ≥1, define
W (zi γ):= max
γ∈{1 γ}M
Fori=1, defineW (zi·)=0. The definition ofM givesM(zi γ)=σ(γ)+W (zi γ)for anyi >1withM(z1·)=0. Note thatW (z2 γ)=W (z1 γ)=0.
DefineW¯—which we will demonstrate is an upper bound and continuous version of W—as
¯
W (zi γ):= ¯σ∗(γ)+ max
γ∈[1γ]W¯
zi−1 γ
+ ¯Wzi−1 γ−γ+1
fori >2with
¯
σ∗(γ):= max
γ∈[1γ]σ¯
γ+ ¯σγ−γ+1 (28)
Fori=1or2, defineW (zi γ)¯ =0. ThenW (zi γ)≤ ¯W (zi γ)for alli≥1and allγ∈Z++. The proof is by induction. They are equal fori=1andi=2. Now consider ani >2and suppose it holds fori−1. Then
W (zi γ)= max
γ∈{1 γ}M
zi−1 γ+Mzi−1 γ−γ+1
= max
γ∈{1 γ}σ
γ+σγ−γ+1+Wzi−1 γ
+Wzi−1 γ−γ+1
≤ max
γ∈{1 γ}σ
γ+σγ−γ+1+ max
γ∈{1 γ}W
zi−1 γ
+Wzi−1 γ−γ+1
≤ max
γ∈{1 γ}σ¯
γ+ ¯σγ−γ+1+ max
γ∈{1 γ} ¯
Wzi−1 γ+ ¯Wzi−1 γ−γ+1
≤ max
γ∈[1γ]σ¯
γ+ ¯σγ−γ+1+ max
γ∈[1γ]W¯
zi−1 γ
+ ¯Wzi−1 γ−γ+1
≤ ¯σ∗(γ)+ max
γ∈[1γ] ¯
Wzi−1 γ
+ ¯Wzi−1 γ−γ+1
= ¯W (zi γ)
Ifσ(x)¯ =xfor allx, thenσ(γ¯ )+ ¯σ(γ−γ+1)=γ+1, which does not depend onγ. So,σ¯∗(γ)=γ+1=2σ(¯ γ+21). If theσ¯ function is strictly increasing, strictly concave, and differentiable, then the first-order condition of theσ¯∗(γ)problem yieldsσ¯(γ)= ¯σ(γ− γ+1). The derivative is invertible (by strict concavity) and soγ= γ+21. So, σ¯∗(γ)=
¯
σ(γ+21)+ ¯σ(γ− γ+21+1), which givesσ¯∗(γ)=2σ(¯ γ+21), the same condition as in the linear case.13Hence,
¯
σ∗(γ)=2σ¯
γ+1 2
(29)
So, fori >2,
¯
W (zi γ)=2σ¯
γ+1
2
+ max
γ∈[1γ] ¯
Wzi−1 γ+ ¯Wzi−1 γ−γ+1 (30)
We will now show for i >2that W (zi γ)¯ =2σ(¯ γ+21)+2W (z¯ i−1γ+21), which gives
a simple recursive relationship for the upper bound (for i =1 or 2, W (zi γ)¯ =0).