Additional Exercises for Convex Optimization
Stephen Boyd Lieven Vandenberghe
March 18, 2016
This is a collection of additional exercises, meant to supplement those found in the book Convex Optimization, by Stephen Boyd and Lieven Vandenberghe. These exercises were used in several courses on convex optimization, EE364a (Stanford), EE236b (UCLA), or 6.975 (MIT), usually for homework, but sometimes as exam questions. Some of the exercises were originally written for the book, but were removed at some point. Many of them include a computational component using CVX, a Matlab package for convex optimization; files required for these exercises can be found at the book web site www.stanford.edu/~boyd/cvxbook/. We are in the process of adapting many of these problems to be compatible with two other packages for convex optimization: CVXPY (Python) and Convex.jl (Julia). Some of the exercises require a knowledge of elementary analysis. You are free to use these exercises any way you like (for example in a course you teach), provided you acknowledge the source. In turn, we gratefully acknowledge the teaching assistants (and in some cases, students) who have helped us develop and debug these exercises. Pablo Parrilo helped develop some of the exercises that were originally used in 6.975.
Course instructors can obtain solutions by email to us. Please specify the course you are teaching and give its URL.
We’ll update this document as new exercises become available, so the exercise numbers and sections will occasionally change. We have categorized the exercises into sections that follow the book chapters, as well as various additional application areas. Some exercises fit into more than one section, or don’t fit well into any section, so we have just arbitrarily assigned these.
Contents
1 Convex sets 3
2 Convex functions 8
3 Convex optimization problems 29
4 Duality 95
5 Approximation and fitting 136
6 Statistical estimation 204
7 Geometry 235
8 Unconstrained and equality constrained minimization 279
9 Interior point methods 298
10 Mathematical background 321
11 Circuit design 324
12 Signal processing and communications 350
13 Finance 375
14 Mechanical and aerospace engineering 455
15 Graphs and networks 513
16 Energy and power 533
1
Convex sets
1.1 Is the set {a ∈ Rk | p(0) = 1, |p(t)| ≤ 1 for α ≤ t ≤ β}, where
p(t) = a1+ a2t +· · · + aktk−1,
convex?
Solution. Yes, it is convex; it is the intersection of an infinite number of slabs,
{a | − 1 ≤ a1+ a2t +· · · + aktk−1 ≤ 1},
parametrized by t∈ [α, β], and the hyperplane
{a | a0= 1}.
1.2 Set distributive characterization of convexity. [?, p21], [?, Theorem 3.2] Show that C ⊆ Rn is
convex if and only if (α + β)C = αC + βC for all nonnegative α, β.
Solution. The equality is trivially true for α = β = 0, so we will assume that α + β 6= 0, and dividing by α + β, we can rephrase the theorem as follows: C is convex if and only if
C = θC + (1− θ)C for 0≤ θ ≤ 1.
C⊆ θC + (1 − θ)C for all θ ∈ [0, 1] is true for any set C, because x = θx + (1 − θ)x. C⊇ θC + (1 − θ)C for all θ ∈ [0, 1] is just a rephrasing of Jensen’s inequality.
1.3 Composition of linear-fractional functions. Suppose φ : Rn → Rm and ψ : Rm → Rp are the
linear-fractional functions
φ(x) = Ax + b
cTx + d, ψ(y) =
Ey + f gTy + h,
with domains dom φ = {x | cTx + d > 0}, dom ψ = {y | gTx + h > 0}. We associate with φ and
ψ the matrices " A b cT d # , " E f gT h # , respectively.
Now consider the composition Γ of ψ and φ, i.e., Γ(x) = ψ(φ(x)), with domain
dom Γ ={x ∈ dom φ | φ(x) ∈ dom ψ}.
Show that Γ is linear-fractional, and that the matrix associated with it is the product
" E f T # " A b T # .
Solution. We have, for x∈ dom Γ,
Γ(x) =E((Ax + b)/(c
Tx + d)) + f
gT(Ax + b)/(cTx + d) + h.
Multiplying numerator and denominator by cTx + d yields
Γ(x) = EAx + Eb + f c Tx + f d gTAx + gTb + hcTx + hd = (EA + f c T)x + (Eb + f d) (gTA + hcT)x + (gTb + hd),
which is the linear fractional function associated with the product matrix.
1.4 Dual of exponential cone. The exponential cone Kexp ⊆ R3 is defined as
Kexp={(x, y, z) | y > 0, yex/y ≤ z}.
Find the dual cone Kexp∗ .
We are not worried here about the fine details of what happens on the boundaries of these cones, so you really needn’t worry about it. But we make some comments here for those who do care about such things.
The cone Kexp as defined above is not closed. To obtain its closure, we need to add the points
{(x, y, z) | x ≤ 0, y = 0, z ≥ 0}.
(This makes no difference, since the dual of a cone is equal to the dual of its closure.) Solution. The dual cone can be expressed as
Kexp∗ = cl{(u, v, w) | u < 0, −uev/u≤ ew}
= {(u, v, w) | u < 0, −uev/u ≤ ew} ∪ {(0, v, w) | v ≥ 0, w ≥ 0},
where cl means closure. We didn’t expect people to add the points on the boundary, i.e., to work out the second set in the second line.
The dual cone can be expressed several other ways as well. The conditions u < 0, −uev/u ≤ ew
can be expressed as
(u/w) log(−u/w) + v/w − u/w ≥ 0, or
u log(−u/w) + v − u ≥ 0, or
log(−u/w) ≤ 1 − (v/u).
Now let’s derive the result. For (u, v, w) to be in Kexp∗ , we need to have ux + vy + wz≥ 0 whenever y > 0 and yex/y ≤ z. Thus, we must have w ≥ 0; otherwise we can make ux + vy + wz negative
by choosing a large enough z. Now let’s see what happens when w = 0. In this case we need ux + vy≥ 0 for all y > 0. This happens only if u = 0 and v ≥ 0. So points with
u = 0, v≥ 0, w = 0 are in Kexp∗ .
Let’s now consider the case w > 0. We’ll minimize ux + vy + wz over all x, y and z that satisfy y > 0 and yex/y ≤ z. Since w > 0, we minimize over z by taking z = yex/y, which yields
ux + vy + wyex/y.
Now we will minimize over x. If u > 0, this is unbounded below as x→ −∞. If u = 0, the minimum is not achieved, but occurs as x → −∞; the minimum value is then vy. This has a nonnegative minimum value over all y > 0 only when v≥ 0. Thus we find that points satisfying
u = 0, v≥ 0, w > 0 are in Kexp∗ .
If u < 0, the minimum of ux + vy + wyex/y occurs when its derivative with respect to x vanishes. This leads to u+wex/y = 0, i.e., x = y log(−u/w). For this value of x the expression above becomes
y(u log(−u/w) + v − u).
Now we minimize over y > 0. We get−∞ if u log(−u/w)+v−u < 0; we get 0 if u log(−u/w)+v−u ≥ 0.
So finally, we have our condition:
u log(−u/w) + v − u ≥ 0, u < 0, w > 0. Dividing by u and taking the exponential, we can write this as
−uev/u≤ ew, u < 0.
(The condition w > 0 is implied by these two conditions.) Finally, then, we have
Kexp∗ ={(u, v, w) | u < 0, −uev/u≤ ew} ∪ {(0, v, w) | v ≥ 0, w ≥ 0}.
1.5 Dual of intersection of cones. Let C and D be closed convex cones in Rn. In this problem we will show that
(C∩ D)∗= C∗+ D∗.
Here, + denotes set addition: C∗+ D∗ is the set {u + v | u ∈ C∗, v ∈ D∗}. In other words, the
dual of the intersection of two closed convex cones is the sum of the dual cones.
(b) Show that (C∩ D)∗ ⊇ C∗+ D∗.
(c) Now let’s show (C∩ D)∗ ⊆ C∗+ D∗. You can do this by first showing (C∩ D)∗⊆ C∗+ D∗ ⇐⇒ C ∩ D ⊇ (C∗+ D∗)∗. You can use the following result:
If K is a closed convex cone, then K∗∗= K.
Next, show that C∩ D ⊇ (C∗+ D∗)∗ and conclude (C∩ D)∗= C∗+ D∗.
(d) Show that the dual of the polyhedral cone V ={x | Ax 0} can be expressed as V∗ ={ATv| v 0}.
Solution.
(a) Suppose x∈ C ∩ D. This implies that x ∈ C and x ∈ D, which implies θx ∈ C and θx ∈ D for any θ≥ 0. Thus, θx ∈ C ∩ D for any θ ≥ 0, which shows C ∩ D is a cone. We know C ∩ D is convex since the intersection of convex sets is convex.
To show C∗+ D∗ is a closed convex cone, note that both C∗ and D∗ are convex cones, thus C∗+ D∗ is the conic hull of C∗∪ D∗, which is a convex cone.
(b) Suppose x ∈ C∗+ D∗. We can write x as x = u + v, where u∈ C∗ and v ∈ D∗. We know
uTy≥ 0 for all y ∈ C and vTy≥ 0 for all y ∈ D, which implies that xTy = uTy + vTy≥ 0 for all y∈ C ∩ D. This shows x ∈ (C ∩ D)∗, and so (C∩ D)∗ ⊇ C∗+ D∗.
(c) We showed in part (a) that C ∩ D and C∗ + D∗ are closed convex cones. This implies (C∩ D)∗∗= C∩ D and (C∗+ D∗)∗∗= (C∗+ D∗), and so
(C∩ D)∗⊆ C∗+ D∗ ⇐⇒ C ∩ D ⊇ (C∗+ D∗)∗.
Suppose x ∈ (C∗+ D∗)∗. xTy ≥ 0 for all y = u + v, where u ∈ C∗, v ∈ D∗. This can be written as xTu + xTv≥ 0, for all u ∈ C∗ and v∈ D∗. Since 0∈ C∗ and 0∈ D∗, taking v = 0
we get xTu≥ 0 for all u ∈ C∗, and taking u = 0 we get xTv ≥ 0 for all v ∈ D∗. This implies x∈ C∗∗= C and x∈ D∗∗= D, i.e., x∈ C ∩ D.
So we have shown both (C ∩ D)∗ ⊇ C∗ + D∗ and (C ∩ D)∗ ⊆ C∗ + D∗, which implies (C∩ D)∗= C∗+ D∗.
(d) Using the result we just proved, we can write
V∗ ={x | aT 1x≥ 0} ∗ +· · · + {x | aT mx≥ 0} ∗ .
The dual of {x | aTi x≥ 0} is the set {θai | θ ≥ 0}, so we get
V∗ = {θa1 | θ ≥ 0} + · · · + {θam | θ ≥ 0}
= {θ1a1+· · · + θmam | θi≥ 0, i = 1, . . . , m}.
This can be more compactly written as
1.6 Polar of a set. The polar of C⊆ Rn is defined as the set
C◦ ={y ∈ Rn| yTx≤ 1 for all x ∈ C}. (a) Show that C◦ is convex (even if C is not).
(b) What is the polar of a cone?
(c) What is the polar of the unit ball for a normk · k?
(d) Show that if C is closed and convex, with 0∈ int C, then (C◦)◦ = C. Solution.
(a) The polar is the intersection of hyperplanes {y | yTx ≤ 1}, parametrized by x ∈ C, so it is convex.
(b) If C is a cone, then we have yTx≤ 1 for all x ∈ C if and only if yTx≤ 0 for all x ∈ C. (To see this, suppose that yTx > 0 for some x∈ C. Then αx ∈ C for all α > 0, so αyTx can be made arbitrarily large, and in particular, exceeds one for α large enough.) Therefore the polar of a cone K is−K∗, the negative of the dual cone.
(c) The polar of the unit ball in the normk · k is the unit ball for the dual norm k · k∗.
(d) Suppose that x ∈ C and y ∈ C◦. Then we have yTx≤ 1. Since this is true for any y ∈ C◦,
we xTy ≤ 1 for all y ∈ C◦. Thus we have x∈ (C◦)◦. So C ⊆ (C◦)◦.
Now suppose that x ∈ (C◦)◦\ C. Then we can find a separating hyperplane for {x} and C, i.e., a6= 0 and b with aTx > b, and aTz≤ b for z ∈ C. Since z = 0 ∈ int C, we have b > 0. By scaling a and b, we can assume that aTx > 1 and aTz ≤ 1 for all z ∈ C. Thus, a ∈ C◦.
Our assumption x∈ (C◦)◦ then tells us that aTx≤ 1, a contradiction. 1.7 Dual cones in R2. Describe the dual cone for each of the following cones.
(a) K ={0}. (b) K = R2.
(c) K ={(x1, x2)| |x1| ≤ x2}.
(d) K ={(x1, x2)| x1+ x2 = 0}.
Solution.
(a) K∗ = R2. To see this:
K∗ = {y | yTx≥ 0 for all x ∈ K}
= {y | yT0≥ 0}
= R2.
(b) K∗ = {0}. To see this, we need to identify the values of y ∈ R2 for which yTx ≥ 0 for all x∈ R2. But given any y6= 0, consider the choice x = −y, for which we have yTx =−kyk22 < 0. So the only possible choice is y = 0 (which indeed satisfies yTx≥ 0 for all x ∈ R2).
(c) K∗ = K. (This cone is self-dual.)
2
Convex functions
2.1 Maximum of a convex function over a polyhedron. Show that the maximum of a convex function f over the polyhedron P = conv{v1, . . . , vk} is achieved at one of its vertices, i.e.,
sup
x∈P
f (x) = max
i=1,...,kf (vi).
(A stronger statement is: the maximum of a convex function over a closed bounded convex set is achieved at an extreme point, i.e., a point in the set that is not a convex combination of any other points in the set.) Hint. Assume the statement is false, and use Jensen’s inequality.
Solution. Let’s assume the statement is false: We have z ∈ P with f(z) > f(vi) for i = 1, . . . , k.
We can represent z as
z = λ1v1+· · · + λkvk,
where λ 0, 1Tλ = 1. Jensen’s inequality tells us that
f (z) = f (λ1v1+· · · + λkvk)
≤ λ1f (v1) +· · · + λkf (vk)
< f (z), so we have a contradiction.
2.2 A general vector composition rule. Suppose
f (x) = h(g1(x), g2(x), . . . , gk(x))
where h : Rk→ R is convex, and gi : Rn→ R. Suppose that for each i, one of the following holds:
• h is nondecreasing in the ith argument, and gi is convex
• h is nonincreasing in the ith argument, and gi is concave
• gi is affine.
Show that f is convex. (This composition rule subsumes all the ones given in the book, and is the one used in software systems such as CVX.) You can assume that dom h = Rk; the result also holds in the general case when the monotonicity conditions listed above are imposed on ˜h, the extended-valued extension of h.
Solution. Fix x, y, and θ ∈ [0, 1], and let z = θx + (1 − θ)y. Let’s re-arrange the indexes so that gi is affine for i = 1, . . . , p, gi is convex for i = p + 1, . . . , q, and gi is concave for i = q + 1, . . . , k.
Therefore we have gi(z) = θgi(x) + (1− θ)gi(y), i = 1, . . . , p, gi(z)≤ θgi(x) + (1− θ)gi(y), i = p + 1, . . . , q, gi(z)≥ θgi(x) + (1− θ)gi(y), i = q + 1, . . . , k. We then have f (z) = h(g1(z), g2(z), . . . , gk(z)) ≤ h(θg1(x) + (1− θ)g1(y), . . . , θgk(x) + (1− θ)gk(y)) ≤ θh(g1(x), . . . , gk(x)) + (1− θ)h(g1(y), . . . , gk(y)) = θf (x) + (1− θ)f(y).
The second line holds since, for i = p + 1, . . . , q, we have increased the ith argument of h, which is (by assumption) nondecreasing in the ith argument, and for i = q + 1, . . . , k, we have decreased the ith argument, and h is nonincreasing in these arguments. The third line follows from convexity of h.
2.3 Logarithmic barrier for the second-order cone. The function f (x, t) =− log(t2−xTx), with dom f =
{(x, t) ∈ Rn× R | t > kxk2} (i.e., the second-order cone), is convex. (The function f is called the
logarithmic barrier function for the second-order cone.) This can be shown many ways, for example by evaluating the Hessian and demonstrating that it is positive semidefinite. In this exercise you establish convexity of f using a relatively painless method, leveraging some composition rules and known convexity of a few other functions.
(a) Explain why t−(1/t)uTu is a concave function on dom f . Hint. Use convexity of the quadratic
over linear function.
(b) From this, show that− log(t − (1/t)uTu) is a convex function on dom f . (c) From this, show that f is convex.
Solution.
(a) (1/t)uTu is the quadratic over linear function, which is convex on dom f . So t− (1/t)uTu is concave, since it is a linear function minus a convex function.
(b) The function g(u) =− log u is convex and decreasing, so if u is a concave (positive) function, the composition rules tell us that g◦ u is convex. Here this means − log(t − (1/t)uTu) is a
convex function on dom f .
(c) We write f (x, t) = − log(t − (1/t)uTu)− log t, which shows that f is a sum of two convex
functions, hence convex.
2.4 A quadratic-over-linear composition theorem. Suppose that f : Rn→ R is nonnegative and convex, and g : Rn→ R is positive and concave. Show that the function f2/g, with domain dom f∩dom g, is convex.
Solution. Without loss of generality we can assume that n = 1. (The general case follows by restricting to an arbitrary line.) Let x and y be in the domains of f and g, and let θ ∈ [0, 1], and define z = θx + (1− θ)y. By convexity of f we have
f (z)≤ θf(x) + (1 − θ)f(y). Since f (x) and θf (x) + (1− θ)f(y) are nonnegative, we have
f (z)2≤ (θf(x) + (1 − θ)f(y))2. (The square function is monotonic on R+.) By concavity of g, we have
g(z)≥ θg(x) + (1 − θ)g(y). Putting the last two together, we have
f (z)2
≤ (θf (x) + (1− θ)f(y))
2
Now we use convexity of the function u2/v, for v > 0, to conclude (θf (x) + (1− θ)f(y))2 θg(x) + (1− θ)g(y) ≤ θ f (x)2 g(x) + (1− θ) f (y)2 g(y) . This, together with the inequality above, finishes the proof.
2.5 A perspective composition rule. [?] Let f : Rn→ R be a convex function with f(0) ≤ 0.
(a) Show that the perspective tf (x/t), with domain{(x, t) | t > 0, x/t ∈ dom f}, is nonincreasing as a function of t.
(b) Let g be concave and positive on its domain. Show that the function
h(x) = g(x)f (x/g(x)), dom h ={x ∈ dom g | x/g(x) ∈ dom f} is convex.
(c) As an example, show that
h(x) = x Tx (Qn k=1xk)1/n , dom h = Rn++ is convex. Solution.
(a) Suppose t2 > t1 > 0 and x/t1 ∈ dom f, x/t2 ∈ dom f. We have x/t2 = θ(x/t1) + (1− θ)0
where θ = t1/t2. Hence, by Jensen’s inequality,
f (x/t2)≤ t1 t2 f (x/t1) + (1− t1 t2 )f (0)≤ t1 t2 f (x/t1) if f (0)≤ 0. Therefore t2f (x/t2)≤ t1f (x/t1).
If we assume that f is differentiable, we can also verify that the derivative of the perspective with respect to t is less than or equal to zero. We have
∂
∂ttf (x/t) = f (x/t)− ∇f(x/t)
T(x/t)
This is less than or equal to zero, because, from convexity of f ,
0≥ f(0) ≥ f(x/t) + ∇f(x/t)T(0− x/t).
(b) This follows from a composition theorem: tf (x/t) is convex in (x, t) and nonincreasing in t; therefore if g(x) is concave then g(x)f (x/g(x)) is convex. This composition rule is actually not mentioned in the lectures or the textbook, but it is easily derived as follows. Let us denote the perspective function tf (x/t) by F (x, t). Consider a convex combination x = θu + (1− θ)v of two points in dom h. Then
h(θu + (1− θ)v) = F (θu + (1 − θ)v, g(θu + (1 − θ)v)) ≤ F (θu + (1 − θ)v, θg(u) + (1 − θ)g(v)),
because g is concave, and F (x, t) is nonincreasing with respect to t. Convexity of F then gives
h(θu + (1− θ)v) ≤ θF (u, g(u)) + (1 − θ)F (v, g(v)) = θh(u) + (1− θ)h(v).
As an alternative proof, we can establish convexity of h directly from the definition. Consider two points u, v ∈ dom g, with u/g(u) ∈ dom f and v/g(v) ∈ dom f. We take a convex combination x = θu + (1− θ)v. This point lies in dom g because dom g is convex. Also,
x g(x) = θg(u) g(x) u g(u) + (1− θ)g(v) g(x) v g(v) = µ1 u g(u) + µ2 v g(v)+ µ3· 0 where µ1 = θg(u) g(x) , µ2 = (1− θ)g(v) g(x) , µ3 = 1− µ1− µ2.
These coefficients are nonnegative and add up to one because g(x) > 0 on its domain, and θg(u) + (1− θ)g(v) ≤ g(x) by concavity of g. Therefore x/g(x) is a convex combination of three points in dom f , and therefore in dom f itself. This shows that dom h is convex. Next we verify Jensen’s inequality:
f (x/g(x)) ≤ µ1f (u/g(u)) + µ2f (v/g(v)) + µ3f (0)
≤ µ1f (u/g(u)) + µ2f (v/g(v))
because f (0)≤ 0. Subsituting the expressions for µ1 and µ2 we get
g(x)f (x/g(x))≤ θg(u)f(u/g(u)) + (1 − θ)g(v)f(v/g(v)) i.e., Jensen’s inequality h(x)≤ θh(u) + (1 − θ)h(v).
(c) Apply part (b) to f (x) = xTx and g(x) = (Q
kxk)1/n.
2.6 Perspective of log determinant. Show that f (X, t) = nt log t−t log det X, with dom f = Sn++×R++,
is convex in (X, t). Use this to show that
g(X) = n(tr X) log(tr X)− (tr X)(log det X) = n n X i=1 λi ! log n X i=1 λi− n X i=1 log λi ! ,
where λi are the eigenvalues of X, is convex on Sn++.
Solution. This is the perspective function of − log det X:
f (X, t) =−t log det(X/t) = nt log t − log det X.
Convexity of g follows from g(X) = f (X, tr X) and the fact that tr X is a linear function of X (and positive).
2.7 Pre-composition with a linear fractional mapping. Suppose f : Rm → R is convex, and A ∈ Rm×n, b∈ Rm, c∈ Rn, and d∈ R. Show that g : Rn→ R, defined by
g(x) = (cTx + d)f ((Ax + b)/(cTx + d)), dom g ={x | cTx + d > 0}, is convex.
Solution. g is just the composition of the perspective of f (which is convex) with the affine map that takes x to (Ax + b, cTx + d), and so is convex.
2.8 Scalar valued linear fractional functions. A function f : Rn→ R is called linear fractional if it has the form f (x) = (aTx + b)/(cTx + d), with dom f ={x | cTx + d > 0}. When is a linear fractional function convex? When is a linear fractional function quasiconvex?
Solution. Linear fractional functions are always quasiconvex, since the sublevel sets are convex, defined by one strict and one nonstrict linear inequality:
f (x)≤ t ⇐⇒ cTx + d > 0, aTx + b− t(cTx + d)≤ 0.
To analyze convexity, we form the Hessian:
∇2f (x) =−(cTx + d)−2(acT + caT) + (aTx + b)(cTx + d)−3ccT.
First assume that a and c are not colinear. In this case, we can find x with cTx + d = 1 (so, x ∈ dom f) with aTx + b taking any desired value. By taking it as a large and negative, we see that the Hessian is not positive semidefinite, so f is not convex.
So for f to be convex, we must have a and c colinear. If c is zero, then f is affine (hence convex). Assume now that c is nonzero, and that a = αc for some α∈ R. In this case, f reduces to
f (x) = αc
Tx + b
cTx + d = α +
b− αd cTx + d,
which is convex if and only if b≥ αd.
So a linear fractional function is convex only in some very special cases: it is affine, or a constant plus a nonnegative constant times the inverse of cTx + d.
2.9 Show that the function
f (x) = kAx − bk
2 2
1− xTx
is convex on {x | kxk2< 1}.
Solution. The epigraph is convex because kAx − bk2
2
t ≤ 1 − x
Tx
defines a convex set.
Here’s another solution: The functionkAx − bk22/u is convex in (x, u), since it is the quadratic over linear function, pre-composed with an affine mapping. This function is decreasing in its second argument, so by a composition rule, we can replace the second argument with any convcave function, and the result is convex. But u = 1 = xTx is concave, so we’re done.
2.10 Weighted geometric mean. The geometric mean f (x) = (Q
kxk)1/n with dom f = Rn++is concave,
as shown on page 74. Extend the proof to show that
f (x) = n Y k=1 xαk k , dom f = R n ++
is concave, where αk are nonnegative numbers with Pkαk= 1.
Solution. The Hessian of f is
∇2f (x) = f (x)qqT − diag(α)−1diag(q)2
where q is the vector (α1/x1, . . . , αn/xn). We have
yT∇2f (x)y = f (x) ( n X k=1 αkyk/xk)2− n X k=1 αkyk2/x2k ! ≤ 0 by applying the Cauchy-Schwarz inequality (uTv)2 ≤ (kuk
2kvk2)2 to the vectors
u = (√α1y1/x1, . . . ,√αnyn/xn), v = (√α1, . . . ,√αn).
2.11 Suppose that f : Rn→ R is convex, and define
g(x, t) = f (x/t), dom g ={(x, t) | x/t ∈ dom f, t > 0}. Show that g is quasiconvex.
Solution. The γ-sublevel set of g is defined by
f (x, t)≤ γ, t > 0, x/t ∈ dom f. This is equivalent to
tf (x, t)≤ tγ, t > 0, x/t ∈ dom f.
The function tf (x/t), with domain {(x, t) | x/t ∈ dom f, t > 0}, is the persepctive function of g, and is convex. The first inequality above is convex, since the righthand side is an affine function of t (for fixed γ). So the γ-sublevel set of g is convex, and we’re done.
2.12 Continued fraction function. Show that the function
f (x) = 1 x1− 1 x2− 1 x3− 1 x4
defined where every denominator is positive, is convex and decreasing. (There is nothing special about n = 4 here; the same holds for any number of variables.)
Solution. We will use the composition rules and recursion. g4(x) = 1/x4 is clearly convex and 1
increasing in z; it follows by the composition rules that g3(x) = x3−g14(x) is convex and decreasing
in all its variables. Repeating this argument for gk(x) = xk−gk+11 (x) shows that f is convex and
decreasing.
Here is another way: g1(x) = x3 − 1/x4 is clearly concave and increasing in x3 and x4. The
function x2− 1/z is concave and increasing in x2 and z; it follows by the composition rules that
g2(x) = x2− 1/g1(x) is concave and increasing. Repeating this argument shows that−f is concave
and increasing, so f is convex and decreasing.
2.13 Circularly symmetric Huber function. The scalar Huber function is defined as
fhub(x) =
(
(1/2)x2 |x| ≤ 1 |x| − 1/2 |x| > 1.
This convex function comes up in several applications, including robust estimation. This prob-lem concerns generalizations of the Huber function to Rn. One generalization to Rn is given by fhub(x1) +· · · + fhub(xn), but this function is not circularly symmetric, i.e., invariant under
transformation of x by an orthogonal matrix. A generalization to Rn that is circularly symmetric is fcshub(x) = fhub(kxk) = ( (1/2)kxk2 2 kxk2 ≤ 1 kxk2− 1/2 kxk2 > 1.
(The subscript stands for ‘circularly symmetric Huber function’.) Show that fcshub is convex. Find
the conjugate function fcshub∗ .
Solution. We can’t directly use the composition form given above, since fhubis not nondecreasing.
But we can write fcshub = h◦ g, where h : R → R and g : Rn→ R are defined as
h(x) = 0 x≤ 0 x2/2 0≤ x ≤ 1 x− 1/2 x > 1,
and g(x) = kxk2. We can think of g as as a version of the scalar Huber function, modified to be
zero when its argument is negative. Clearly, g is convex and ˜h is convex and increasing. Thus, from the composition rules we conclude that fcshub is convex.
Now we will show that
fcshub∗ (y) =
(
(1/2)kyk2
2 kyk2 ≤ 1
∞ otherwise. Supposekyk2 > 1. Taking x = ty/kyk2, we see that for t≥ 1 we have
yTx− f(x) = tkyk2− t + 1/2 = t(kyk2− 1) + 1/2.
Letting t→ ∞, we see that for any y with kyk2> 1, supx(yTx− f(x)) = ∞. Thus, fcshub∗ (y) =∞
forkyk2 > 1.
Now suppose kyk2≤ 1. We can write fcshub∗ (y) as
fcshub∗ (y) = max
( sup kxk2≤1 (yTx− (1/2)kxk22), sup kxk2≥1 (yTx− kxk2+ 1/2) ) .
It is easy to show that yTx− (1/2)kxk22 is maximized over {x | kxk2 ≤ 1} when x = y (set the
gradient of yTx− (1/2)kxk2
2 equal to zero). This gives
sup
kxk2≤1
(yTx− (1/2)kxk2
2) = (1/2)kyk22.
To find supkxk2≥1(yTx− kxk2+ 1/2), notice that forkxk2 ≥ 1
yTx− kxk2+ 1/2≤ kyk2kxk2− kxk2+ 1/2 =kxk2(kyk2− 1) + 1/2 ≤ kyk2− 1/2.
Here, the first inequality follows from Cauchy-Schwarz, and the second inequality follows from kyk2 ≤ 1 and kxk2 ≥ 1. Furthermore, if we choose x = y/kyk2, then
yTx− kxk2+ 1/2 =kyk2− 1/2, thus, sup kxk2≥1 (yTx− kxk2+ 1/2) =kyk2− 1/2. Forkyk2 ≤ 1 sup kxk2≥1
(yTx− kxk2+ 1/2) =kyk2− 1/2 ≤ (1/2)kyk22 = sup kxk2≤1
(yTx− (1/2)kxk22), so we conclude that forkyk2 ≤ 1, fcshub∗ (y) = (1/2)kyk22.
2.14 Reverse Jensen inequality. Suppose f is convex, λ1> 0, λi ≤ 0, i = 2, . . . , k, and λ1+· · · + λn= 1,
and let x1, . . . , xn∈ dom f. Show that the inequality
f (λ1x1+· · · + λnxn)≥ λ1f (x1) +· · · + λnf (xn)
always holds. Hints. Draw a picture for the n = 2 case first. For the general case, express x1 as a
convex combination of λ1x1+· · · + λnxn and x2, . . . , xn, and use Jensen’s inequality.
Solution. Let z = λ1x1+· · · + λnxn, with λ1 > 0, λi ≤ 0, i = 2, . . . , n, and λ1+· · · + λn = 1.
Then we have
x1 = θ0z + θ2x2+· · · + θnxn,
where
θi =−λi/λ1, i = 2, . . . , n,
and θ0 = 1/λ1. Since λ1 > 1, we see that θ0 > 0; from λi≤ 0 we get θi≥ 0 for i = 2, . . . , n. Simple
algebra shows that θ1+· · · + θn= 1. From Jensen’s inequality we have
f (x1)≤ θ0f (z) + θ2f (x2) +· · · + θnf (xn), so f (z)≥ 1 θ0 f (x1) + θ2 θ0 f (x2) +· · · + θn θ0 f (xn),
Substituting for θi this becomes the inequality we want,
2.15 Monotone extension of a convex function. Suppose f : Rn→ R is convex. Recall that a function h : Rn → R is monotone nondecreasing if h(x) ≥ h(y) whenever x y. The monotone extension of f is defined as
g(x) = inf
z0f (x + z).
(We will assume that g(x) > −∞.) Show that g is convex and monotone nondecreasing, and satisfies g(x) ≤ f(x) for all x. Show that if h is any other convex function that satisfies these properties, then h(x)≤ g(x) for all x. Thus, g is the maximum convex monotone underestimator of f .
Remark. For simple functions (say, on R) it is easy to work out what g is, given f . On Rn, it can be very difficult to work out an explicit expression for g. However, systems such as CVX can immediately handle functions such as g, defined by partial minimization.
Solution. The function f (x + z) is jointly convex in x and z; partial minimization over z in the nonnegative orthant yields g, which therefore is convex.
To show that g is monotone, suppose that x y. Then we have x = y + ˆz, where ˆz = x − y 0, and so
g(y) = inf
z0f (y + z)≤ f(y + ˆz) = f(x).
To how that g(x)≤ f(x), we use
g(x) = inf
z0f (x + z)≤ f(x).
Now suppose that h is monotone, convex, and satisfies h(x)≤ f(x) for all x. Then we have, for all x and z, h(x + z)≤ f(x + z). Taking the infimum over z 0, we obtain
inf
z0h(x + z)≤ infz0f (x + z).
Since h is monotone nondecreasing, the lefthand side is h(x); the righthand side is g(x).
2.16 Circularly symmetric convex functions. Suppose f : Rn→ R is convex and symmetric with respect to rotations, i.e., f (x) depends only on kxk2. Show that f must have the form f (x) = φ(kxk2),
where φ : R→ R is nondecreasing and convex, with dom f = R. (Conversely, any function of this form is symmetric and convex, so this form characterizes such functions.)
Solution. Define ψ(a) = f (ae1), where e1 is the first standard unit vector, and a ∈ R. ψ is a
convex function, and it is symmetric: ψ(−a) = f(−ae1) = f (ae1) = ψ(a), sincek − ae1k2=kae1k2.
A symmetric convex function on R must have its minimum at 0; for suppose that ψ(a) < ψ(0). By Jensen’s inequality, ψ(0) ≤ (1/2)ψ(−a) + (1/2)ψ(a) = ψ(a), a contradiction. Therefore ψ is nondecreasing for a ≥ 0. Now we define φ(a) = ψ(a) for a ≥ 0 and φ(a) = ψ(0) for a < 0. ψ is convex and nondecreasing. Evidently we have f (x) = φ(kxk2), so we’re done.
2.17 Infimal convolution. Let f1, . . . , fm be convex functions on Rn. Their infimal convolution, denoted
g = f1 · · · fm (several other notations are also used), is defined as
with the natural domain (i.e., defined by g(x) <∞). In one simple interpretation, fi(xi) is the cost
for the ith firm to produce a mix of products given by xi; g(x) is then the optimal cost obtained
if the firms can freely exchange products to produce, all together, the mix given by x. (The name ‘convolution’ presumably comes from the observation that if we replace the sum above with the product, and the infimum above with integration, then we obtain the normal convolution.)
(a) Show that g is convex. (b) Show that g∗= f1∗+· · · + f∗
m. In other words, the conjugate of the infimal convolution is the
sum of the conjugates.
Solution.
(a) We can express g as
g(x) = inf
x1,...,xm
(f1(x1) +· · · + fm(xm) + φ(x1, . . . , xm, x)) ,
where φ(x1, . . . , xm, x) is 0 when x1 +· · · + xm = x, and ∞ otherwise. The function on the
righthand side above is convex in x1, . . . , xm, x, so by the partial minimization rule, so is g.
(b) We have g∗(y) = sup x (yTx− f(x)) = sup x yTx− inf x1+···+xm=x f1(x1) +· · · + fm(xm) = sup x=x1+···+xm yTx1− f1(x1) +· · · + yTxm− fm(xm) ,
where we use the fast that (− inf S) is the same as (sup −S). The last line means we are to take the supremum over all x and all x1, . . . , xm that sum to x. But this is the same as just
taking the supremum over all x1, . . . , xm, so we get
g∗(y) = sup x1,...,xm yTx1− f1(x1) +· · · + yTxm− fm(xm) = sup x1 (yTx1− f1(x1)) +· · · + sup xm (yTxm− fm(xm)) = f1∗(y) +· · · + fm∗(y).
2.18 Conjugate of composition of convex and linear function. Suppose A ∈ Rm×n with rank A = m, and g is defined as g(x) = f (Ax), where f : Rm→ R is convex. Show that
g∗(y) = f∗((A†)Ty), dom(g∗) = ATdom(f∗),
where A†= (AAT)−1A is the pseudo-inverse of A. (This generalizes the formula given on page 95 for the case when A is square and invertible.)
Solution. Let z = (A†)Ty, so y = ATz. Then we have
g?(y) = sup
x (y Tx
− f(Ax)) = sup(zT(Ax)− f(Ax))
= sup w (z Tw − f(w)) = f∗(z) = f∗((A†)Ty).
Now y ∈ dom(g∗) if and only if (A†)Ty = z ∈ dom(f∗). But since y = ATz, we see that this is
equivalent to y∈ ATdom(f∗).
2.19 [?, p104] Suppose λ1, . . . , λn are positive. Show that the function f : Rn→ R, given by
f (x) = n Y i=1 (1− e−xi)λi, is concave on dom f = ( x∈ Rn++ n X i=1 λie−xi ≤ 1 ) .
Hint. The Hessian is given by
∇2f (x) = f (x)(yyT − diag(z))
where yi = λie−xi/(1− e−xi) and zi = yi/(1− e−xi).
Solution. We’ll use Cauchy-Schwarz to show that the Hessian is negative semidefinite. The Hessian is given by
∇2f (x) = f (x) (yyT − diag(z))
where yi = λie−xi/(1− e−xi) and zi = yi2/(λie−xi).
For any v∈ Rn, we can show
vT∇2f (x)v = n X i=1 viyi !2 − n X i=1 vi2yi2 λie−xi ≤ 0,
by applying the Cauchy-Schwarz inequality, (aTb)2 ≤ (aTa)(bTb), to the vectors with components ai = viyi/
√
λie−xi and bi =
√
λie−xi. The result follows because (bTb) ≤ 1 on dom f. Thus the
Hessian is negative semidefinite.
2.20 Show that the following functions f : Rn→ R are convex.
(a) The difference between the maximum and minimum value of a polynomial on a given interval, as a function of its coefficients:
f (x) = sup t∈[a,b] p(t)− inf t∈[a,b]p(t) where p(t) = x1+ x2t + x3t 2+· · · + x ntn−1.
a, b are real constants with a < b.
(b) The ‘exponential barrier’ of a set of inequalities:
f (x) = m X i=1 e−1/fi(x), dom f ={x | f i(x) < 0, i = 1, . . . , m}.
(c) The function
f (x) = inf
α>0
g(y + αx)− g(y) α
if g is convex and y ∈ dom g. (It can be shown that this is the directional derivative of g at y in the direction x.)
Solution.
(a) f is the difference of a convex and a concave function. The first term is convex, because it is the supremum of a family of linear functions of x. The second term is concave because it is the infimum of a family of linear functions.
(b) h(u) = exp(1/u) is convex and decreasing on R++:
h0(u) =− 1 u2e 1/u, h00 (u) = 2 u3e 1/u+ 1 u4e 1/u.
Therefore the composition h(−fi(x)) = exp(−1/fi(x)) is convex if fi is convex.
(c) Can be written as f (x) = inf t>0t g(y +1 tx)− g(y) ,
the infimum over t of the perspective of the convex function g(y + x)− g(y)).
2.21 Symmetric convex functions of eigenvalues. A function f : Rn→ R is said to be symmetric if it is invariant with respect to a permutation of its arguments, i.e., f (x) = f (P x) for any permutation matrix P . An example of a symmetric function is f (x) = log(Pn
k=1exp xk).
In this problem we show that if f : Rn→ R is convex and symmetric, then the function g : Sn→ R
defined as g(X) = f (λ(X)) is convex, where λ(X) = (λ1(X), λ2(x), . . . , λn(X)) is the vector of
eigenvalues of X. This implies, for example, that the function
g(X) = log tr eX = log n X k=1 eλk(X) is convex on Sn.
(a) A square matrix S is doubly stochastic if its elements are nonnegative and all row sums and column sums are equal to one. It can be shown that every doubly stochastic matrix is a convex combination of permutation matrices.
Show that if f is convex and symmetric and S is doubly stochastic, then f (Sx)≤ f(x).
(b) Let Y = Q diag(λ)QT be an eigenvalue decomposition of Y ∈ Sn with Q orthogonal. Show
that the n× n matrix S with elements Sij = Q2ij is doubly stochastic and that diag(Y ) = Sλ.
(c) Use the results in parts (a) and (b) to show that if f is convex and symmetric and X ∈ Sn,
then
f (λ(X)) = sup
V ∈V
f (diag(VTXV ))
Solution.
(a) Suppose S is expressed as a convex combination of permutation matrices: S =X
k
θkPk
with θk≥ 0, Pkθk = 1, and Pk permutation matrices. From convexity and symmetry of f ,
f (Sx) = f (X k θkPkx)≤ X k θkf (Pkx) = X k θkf (x) = f (x). (b) From X = Q diag(λ)QT, Xii= n X j=1 Q2ijλj. From QQT = I, we have P jQ2ij = 1. From QTQ = I, we have P iQ2ij = 1.
(c) Combining the results in parts (a) and (b), we conclude that for any symmetric X, the inequality
f (diag(X))≤ f(λ(X))
holds. Moreover, if V is orthogonal, then λ(X) = λ(VTXV ). Therefore also f (diag(VTXV ))≤ f(λ(X))
for all orthogonal V , with equality if V = Q. In other words f (λ(X)) = sup
V ∈V
f (diag(VTXV )).
This shows that f (λ(X)) is convex because it is the supremum of a family of convex functions of X.
2.22 Convexity of nonsymmetric matrix fractional function. Consider the function f : Rn×n× Rn→ R, defined by
f (X, y) = yTX−1y, dom f ={(X, y) | X + XT 0}. When this function is restricted to X∈ Sn, it is convex.
Is f convex? If so, prove it. If not, give a (simple) counterexample.
Solution. The function is not convex. Restrict the function f to g(s) = f (X, y), with
X = " 1 s −s 1 # , y = " 1 1 # ,
and s∈ R. (The domain of g is R.) If f is convex then so is g. But we have g(s) = 2
1 + s2,
which is certainly not convex.
For a very specific counterexample, take (say) s = +1 and s = −1. Then we have g(−1) = 1, g(+1) = 1 and
2.23 Show that the following functions f : Rn→ R are convex.
(a) f (x) =− exp(−g(x)) where g : Rn→ R has a convex domain and satisfies
" ∇2g(x) ∇g(x) ∇g(x)T 1 # 0 for x∈ dom g. (b) The function
f (x) = max{kAP x − bk | P is a permutation matrix} with A∈ Rm×n, b∈ Rm.
Solution.
(a) The gradient and Hessian of f are
∇f(x) = e−g(x)∇g(x) ∇2f (x) = e−g(x) ∇2g(x)− e−g(x) ∇g(x)∇g(x)T = e−g(x)∇2g(x)− ∇g(x)∇g(x)T 0.
(b) f is the maximum of convex functionskAP x − bk, parameterized by P .
2.24 Convex hull of functions. Suppose g and h are convex functions, bounded below, with dom g = dom h = Rn. The convex hull function of g and h is defined as
f (x) = inf{θg(y) + (1 − θ)h(z) | θy + (1 − θ)z = x, 0 ≤ θ ≤ 1} ,
where the infimum is over θ, y, z. Show that the convex hull of h and g is convex. Describe epi f in terms of epi g and epi h.
Solution. We note that f (x)≤ t if and only if there exist θ ∈ [0, 1], y, z, t1, t2 such that
g(y)≤ t1, h(z)≤ t2, θy + (1− θ)z = x, θt1+ (1− θ)t2= t.
Thus
epi f = conv (epi g∪ epi h) ,
i.e., epi f is the convex hull of the union of the epigraphs of g and h. This shows that f is convex. As an alternative proof, we can make a change of variable ˜y = θy, ˜z = (1− θ)z in the minimization problem in the definition, and note that f (x) is the optimal value of
minimize θg(y/θ) + (1− θ)h(z/(1 − θ))) subject to y + z = x
0≤ θ ≤ 1,
with variables θ, y, z. This is a convex problem, and therefore the optimal value is a convex function of the righthand side x.
2.25 Show that a function f : R→ R is convex if and only if dom f is convex and det 1 1 1 x y z f (x) f (y) f (z) ≥ 0
for all x, y, z ∈ dom f with x < y < z. Solution. det 1 1 1 t1 t2 t3 f (t1) f (t2) f (t3) = det 1 1 1 t1 t2 t3 f (t1) f (t2) f (t3) 1 −1 0 0 1 −1 0 0 1 = 1 0 0 t1 t2− t1 t3− t2 f (t1) f (t2)− f(t1) f (t3)− f(t2) = (t2− t1)(f (t3)− f(t2))− (t3− t2)(f (t2)− f(t1)).
This is nonnegative if and only if t3− t1 (t2− t1)(t3− t2) f (t2)≤ 1 t2− t1 f (t1) + 1 t3− t2 f (t3).
This is Jensen’s inequality
f (θt1+ (1− θ)t3)≤ θf(t1) + (1− θ)f(t3) with θ = t2− t1 t3− t1 , 1− θ = t3− t2 t3− t1 .
2.26 Generalization of the convexity of log det X−1. Let P ∈ Rn×m have rank m. In this problem we show that the function f : Sn→ R, with dom f = Sn++, and
f (X) = log det(PTX−1P )
is convex. To prove this, we assume (without loss of generality) that P has the form
P = " I 0 # ,
where I. The matrix PTX−1P is then the leading m× m principal submatrix of X−1.
(a) Let Y and Z be symmetric matrices with 0≺ Y Z. Show that det Y ≤ det Z. (b) Let X ∈ Sn++, partitioned as X = " X11 X12 X12T X22 # ,
with X11∈ Sm. Show that the optimization problem
minimize log det Y−1
subject to " Y 0 0 0 # " X11 X12 X12T X22 # ,
with variable Y ∈ Sm, has the solution
Y = X11− X12X22−1X T 12.
(As usual, we take Sm++ as the domain of log det Y−1.)
Hint. Use the Schur complement characterization of positive definite block matrices (page 651 of the book): if C 0 then
" A B BT C # 0 if and only if A− BC−1BT 0.
(c) Combine the result in part (b) and the minimization property (page 3-19, lecture notes) to show that the function
f (X) = log det(X11− X12X22−1X T 12)−1,
with dom f = Sn++, is convex.
(d) Show that (X11− X12X22−1X12T)−1 is the leading m× m principal submatrix of X−1, i.e.,
(X11− X12X22−1X12T)−1 = PTX−1P.
Hence, the convex function f defined in part (c) can also be expressed as f (X) = log det(PTX−1P ). Hint. Use the formula for the inverse of a symmetric block matrix:
" A B BT C #−1 = " 0 0 0 C−1 # + " −I C−1BT # (A− BC−1BT)−1 " −I C−1BT #T
if C and A− BC−1BT are invertible. Solution.
(a) Y Z if and only if Y−1/2ZY−1/2 0, which implies det(Y−1/2ZY−1/2) = det Y−1det Z =
det Z/ det Y ≥ 0.
(b) The optimal Y maximizes det Y subject to the constraint
" X11 X12 X12T X22 # − " Y 0 0 0 # = " X11− Y X12 X12T X22 # 0.
By the Schur complement property in the hint this inequality holds if and only if
Y X11− X12X22−1X12T
(c) Define a function F : Sn× Sm → R with F (X, Y ) = log det Y−1 on the domain dom F = ( (X, Y ) Y 0, " X11 X12 X12T X22 # " Y 0 0 0 # ) .
This function is convex because its domain is convex and log det Y−1 is convex on the set of positive definite Y . In part (b) we proved that
f (X) = inf
Y F (X, Y )
and convexity follows from the minimization property.
(d) The formula for the inverse shows that (A− BC−1BT)−1 is the 1,1 block of the inverse of the block matrix.
2.27 Functions of a random variable with log-concave density. Suppose the random variable X on Rn has log-concave density, and let Y = g(X), where g : Rn→ R. For each of the following statements, either give a counterexample, or show that the statement is true.
(a) If g is affine and not constant, then Y has log-concave density. (b) If g is convex, then prob(Y ≤ a) is a log-concave function of a.
(c) If g is concave, then E ((Y − a)+) is a convex and log-concave function of a. (This quantity is
called the tail expectation of Y ; you can assume it exists. We define (s)+as (s)+= max{s, 0}.)
Solution.
(a) This one is true. Let p be the density of X, and let g(x) = cTx + d, with c6= 0 (otherwise g would be constant). Since g is not constant, we conclude that Y has a density pY.
With δa > 0, define the function
h(x, a) =
(
1 a≤ g(x) ≤ a + δa 0 otherwise,
which is the 0− 1 indicator function of the convex set {(x, a) | a ≤ g(x) ≤ a + δa}. The 0 − 1 indicator function of a convex set is log-concave, so by the integration rule it follows that
Z
p(x)h(x, a) dx = E h(X, a) = prob(a≤ Y ≤ a + δa) is log-concave in a. It follows that
prob(a≤ Y ≤ a + δa) δa
is log-concave (since δa > 0). Taking δa → 0, this converges to pY(a), which we conclude is
log-concave.
(b) This one is true. Here we define the function
h(x, a) =
(
1 g(x)≤ a 0 otherwise,
which is the 0−1 indicator function of the convex set epi g = {(x, a) | g(x) ≤ a}, and therefore log-concave. By the integration rule we get that
Z
p(x)h(x, a) dx = E h(X, a) = prob(Y ≤ a) is log-concave in a.
If we assume that g is concave, and we switch the inequality, we conclude that prob(Y ≥ a) is log-concave in a. (We’ll use this below.)
(c) This one is true. Convexity of the tail expectation holds for any random variable, so it has has nothing to do with g, and it has nothing to do with log-concavity of the density of X. For any random variable Y on R, we have
d
daE(Y − a)+=− prob(Y ≥ a).
The righthand side is nondecreasing in a, so the tail expectation has nondecreasing derivative, which means it is a convex function.
Now let’s show that the tail expectation is log-concave. One simple method is to use the formula above to note that
E(Y − a)+=
Z ∞
a
prob(Y ≥ b) db.
The integration rule for log-concave functions tells us that this is log-concave.
We can also give a direct proof following the style of the ones given above. We define g as h(x, a) = (g(x)− a)+. This function is log-concave. First, its domain is {(x, a) | g(x) > a},
which is convex. Concavity of log h(x, a) = log(g(x)− a) follows from the composition rule: log is concave and increasing, and g(x)− a is concave in (x, a). So by the integration rule we get
Z
p(x)h(x, a) dx = E(g(x)− a)+
is log-concave in a.
2.28 Majorization. Define C as the set of all permutations of a given n-vector a, i.e., the set of vectors (aπ1, aπ2, . . . , aπn) where (π1, π2, . . . , πn) is one of the n! permutations of (1, 2, . . . , n).
(a) The support function of C is defined as SC(y) = maxx∈CyTx. Show that
SC(y) = a[1]y[1]+ a[2]y[2]+· · · + a[n]y[n].
(u[1], u[2], . . . , u[n] denote the components of an n-vector u in nonincreasing order.) Hint. To find the maximum of yTx over x∈ C, write the inner product as
yTx = (y1− y2)x1+ (y2− y3)(x1+ x2) + (y3− y4)(x1+ x2+ x3) +· · ·
+ (yn−1− yn)(x1+ x2+· · · + xn−1) + yn(x1+ x2+· · · + xn)
(b) Show that x satisfies xTy≤ SC(y) for all y if and only if
sk(x)≤ sk(a), k = 1, . . . , n− 1, sn(x) = sn(a),
where skdenotes the function sk(x) = x[1]+ x[2]+· · · + x[k]. When these inequalities hold, we
say the vector a majorizes the vector x.
(c) Conclude from this that the conjugate of SC is given by
S∗C(x) =
(
0 if x is majorized by a +∞ otherwise.
Since SC∗ is the indicator function of the convex hull of C, this establishes the following result: x is a convex combination of the permutations of a if and only if a majorizes x.
Solution.
(a) Suppose y is sorted. From the expression of the inner product it is clear that the permutation of a that maximizes the inner product with y is xk = a[k], k = 1, . . . , n.
(b) We first show that if a majorizes x, then xTy ≤ SC(y) for all y. Note that if x is majorized
by a, then all permutations of x are majorized by a, so we can assume that the components of y are sorted in nonincreasing order. Using the results from part a,
SC(y)− xTy = (y1− y2)(s1(a)− x1) + (y2− y3)(s2(a)− x1− x2) +· · · + (yn−1− yn)(sn−1(a)− x1− · · · − xn−1) + yn(sn(a)− x1− · · · − xn) ≥ (y1− y2)(s1(a)− s1(x)) + (y2− y3)(s2(a)− s2(x)) +· · · + (yn−1− yn)(sn−1(a)− sn−1(x)) + yn(sn(a)− sn(x)) ≥ 0.
Next, we show that the conditions are necessary. We distinguish two cases.
• Suppose sk(x) > sk(a) for some k < n. Assume the components of x are sorted in
nonincreasing order and choose
y1=· · · = yk−1= 1, yk =· · · = yn= 0.
Then SC(y)− xTy = sk(a)− sk(x) < 0.
• Suppose sn(x)6= sn(a). Choose y = 1 if sn(x) > sn(a) and y =−1 if sn(x) < sn(a). We
have SC(y)− xTy = y1(sn(a)− sn(x)) < 0.
(c) The expression for the conjugate follows from the fact that if SC(y)− xTy is positive for some
y then it is unbounded above, and if x is majorized by a then x = 0 is the optimum.
2.29 Convexity of products of powers. This problem concerns the product of powers function f : Rn++→ R given by f (x) = xθ1
1 · · · xθnn, where θ ∈ Rn is a vector of powers. We are interested in finding
values of θ for which f is convex or concave. You already know a few, for example when n = 2 and θ = (2,−1), f is convex (the quadratic-over-linear function), and when θ = (1/n)1, f is concave
(geometric mean). Of course, if n = 1, f is convex when θ ≥ 1 or θ ≤ 0, and concave when 0≤ θ ≤ 1.
Show each of the statements below. We will not read long or complicated proofs, or ones that involve Hessians. We are looking for short, snappy ones, that (where possible) use composition rules, perspective, partial minimization, or other operations, together with known convex or concave functions, such as the ones listed in the previous paragraph. Feel free to use the results of earlier statements in later ones.
(a) When n = 2, θ 0, and 1Tθ = 1, f is concave.
(b) When θ 0 and 1Tθ = 1, f is concave. (This is the same as part (a), but here it is for general n.)
(c) When θ 0 and 1Tθ≤ 1, f is concave.
(d) When θ 0, f is convex.
(e) When 1Tθ = 1 and exactly one of the elements of θ is positive, f is convex. (f) When 1Tθ≥ 1 and exactly one of the elements of θ is positive, f is convex.
Remark. Parts (c), (d), and (f) exactly characterize the cases when f is either convex or concave. That is, if none of these conditions on θ hold, f is neither convex nor concave. Your teaching staff has, however, kindly refrained from asking you to show this.
Solution. To shorten our proofs, when both x and θ are vectors, we overload notation so that
f (x) = xθ1
1 · · · xθnn = xθ.
(a) Since xθ1
1 is concave for 0≤ θ1 ≤ 1, applying the perspective transformation gives that
x2(x1/x2)θ1 = xθ11x 1−θ1 2
is concave, which is what we wanted.
(b) The proof is by induction on n. We know the base case with n = 1 holds. For the induction step, if θ∈ Rn+1+ , ˜θ = (θ1, . . . , θn), ˜x = (x1, . . . , xn), and 1Tθ = 1, then ˜xθ/1˜
Tθ˜
is concave by the induction assumption. The function y1Tθ˜z1−1Tθ˜is concave by (a) and nondecreasing. The composition rules give that
(˜xθ/1˜ Tθ)1Tθ˜xn+11−1Tθ˜= ˜xθ˜xθn+1 n+1 = x
θ
is concave.
(c) If 1Tθ ≤ 1, then xθ/1Tθ
is concave by (b). The function y1Tθ is concave and nondecreasing. Composition gives that
(xθ/1Tθ)1Tθ = xθ is concave.
(d) If θ 0, then xθ/1Tθ
is concave by part (b). (We can assume 1Tθ6= 0.) The function y1Tθ
is convex and nonincreasing, since 1Tθ < 0. Composition gives that
is convex.
Here’s another proof, that several people used, and which is arguably simpler than the one above. Since θi ≤ 0, θilog xi is a convex function of xi, and therefore the sum Piθilog xi is
convex in x. By the composition rules, the exponential of a convex function is convex, so
exp(X
i
θilog xi) = xθ
is convex.
(e) If θ ∈ Rn+1 and 1Tθ = 1, we can assume that the single positive element is θ
n+1 > 0, so
that ˜θ = (θ1, . . . , θn) 0. If ˜x = (x1, . . . , xn), then ˜x ˜
θ is convex by part (d). Applying the
perspective transformation gives that
xn+1(˜x/xn+1) ˜ θ = ˜xθ˜x1−1n+1Tθ˜= ˜xθ˜xθn+1 n+1 = xθ is convex.
(f) If 1Tθ ≥ 1 and exactly one element of θ is positive, then xθ/1Tθ
is convex by part (e). The function y1Tθ is convex and nondecreasing. Composition gives us that
(xθ/1Tθ)1Tθ = xθ
is convex.
Remark. The proofs for (c), (d), and (f) are syntactically identical.
Remark. We can also prove (c) with the following self-contained argument. A syntactically identical self-contained argument also works for (f) by substituting “convex” for “concave”.
The proof is by induction on n. We know the base case: xθ1
1 is concave for 0 ≤ θ1 ≤ 1. For the
inductive step, if θ ∈ Rn+1+ and 1Tθ ≤ 1, let ˜θ = (θ1, . . . , θn) and ˜x = (x1, . . . , xn). Note that
˜
xθ/1˜ Tθ is concave by the induction assumption. Applying the perspective transformation gives that
xn+1(˜x/xn+1) ˜ θ/1Tθ
= ˜xθ/1˜ Tθx1−1n+1Tθ/1˜ Tθ
is concave. The function y1Tθ is concave and nondecreasing, and composing it with the previous function shows that
(˜xθ/1˜ Tθxn+11−1Tθ/1˜ Tθ)1Tθ= ˜xθ˜x1n+1Tθ−1Tθ˜= ˜xθ˜xθn+1 n+1 = xθ
3
Convex optimization problems
3.1 Minimizing a function over the probability simplex. Find simple necessary and sufficient conditions for x∈ Rn to minimize a differentiable convex function f over the probability simplex,{x | 1Tx = 1, x 0}.
Solution. The simple basic optimality condition is that x is feasible, i.e., x 0, 1Tx = 1, and that∇f(x)T(y− x) ≥ 0 for all feasible y. We’ll first show this is equivalent to
min
i=1,...,n∇f(x)i ≥ ∇f(x) Tx.
To see this, suppose that ∇f(x)T(y − x) ≥ 0 for all feasible y. Then in particular, for y = e i,
we have ∇f(x)i ≥ ∇f(x)Tx, which is what we have above. To show the other way, suppose that
∇f(x)i ≥ ∇f(x)Tx holds, for i = 1, . . . , n. Let y be feasible, i.e., y 0, 1Ty = 1. Then multiplying
∇f(x)i ≥ ∇f(x)Tx by yi and summing, we get n X i=1 yi∇f(x)i ≥ n X i=1 yi ! ∇f(x)Tx =∇f(x)Tx. The lefthand side is yT∇f(x), so we have ∇f(x)T(y− x) ≥ 0.
Now we can simplify even further. The condition above can be written as
min i=1,...,n ∂f ∂xi ≥ n X i=1 xi ∂f ∂xi .
But since 1Tx = 1, x 0, we have
min i=1,...,n ∂f ∂xi ≤ n X i=1 xi ∂f ∂xi ,
and it follows that
min i=1,...,n ∂f ∂xi = n X i=1 xi ∂f ∂xi .
The right hand side is a mixture of ∂f /∂xi terms and equals the minimum of all of the terms. This
is possible only if xk= 0 whenever ∂f /∂xk> mini∂f /∂xi.
Thus we can write the (necessary and sufficient) optimality condition as 1Tx = 1, x 0, and, for each k, xk> 0 ⇒ ∂f ∂xk = min i=1,...,n ∂f ∂xi .
In particular, for k’s with xk> 0, ∂f /∂xk are all equal.
3.2 ‘Hello World’ in CVX*. Use CVX, CVX.PY or Convex.jl to verify the optimal values you obtained (analytically) for exercise 4.1 in Convex Optimization.
Solution.
(b) p? =−∞ (c) p? = 0
(d) p? = 13 (e) p? = 12
%exercise 4.1 using CVX
%set up a vector to store optimal values of problems optimal_values=zeros(5,1); %part a cvx_begin variable x(2) minimize(x(1)+x(2)) 2*x(1)+x(2) >= 0; x(1)+3*x(2) >= 1; x >= 0; cvx_end optimal_values(1)=cvx_optval; %part b cvx_begin variable x(2) minimize(-sum(x)) 2*x(1)+x(2) >= 0; x(1)+3*x(2) >= 1; x >= 0; cvx_end optimal_values(2)=cvx_optval; %part c cvx_begin variable x(2) minimize(x(1)) 2*x(1)+x(2) >= 0; x(1)+3*x(2) >= 1; x >= 0; cvx_end optimal_values(3)=cvx_optval; %part d cvx_begin variable x(2) minimize(max(x))
2*x(1)+x(2) >= 0; x(1)+3*x(2) >= 1; x >= 0; cvx_end optimal_values(4)=cvx_optval; %part e cvx_begin variable x(2,1)
minimize( square(x(1))+ 9*square(x(2)) ) 2*x(1)+x(2) >= 0; x(1)+3*x(2) >= 1; x >= 0; cvx_end optimal_values(5)=cvx_optval; import cvxpy as cvx import numpy as np x1 = cvx.Variable() x2 = cvx.Variable() constraints = [2*x1 + x2 >= 1, x1 + 3*x2 >= 1, x1 >= 0, x2 >= 0] prob = cvx.Problem([], constraints)
# (a)
prob.objective = cvx.Minimize(x1+x2) prob.solve()
print ’With obj. x1+x2, p* is %.2f, optimal x1 is %.2f, optimal x2 is %.2f’ \ % (prob.value, x1.value, x2.value)
# (b)
prob.objective = cvx.Minimize(-x1-x2) prob.solve(solver=cvx.CVXOPT)
print ’With obj. -x1-x2, status is ’ + prob.status
# (c)
prob.objective = cvx.Minimize(x1) prob.solve()
print ’With obj. x1, p* is %.2f, x1* is %.2f, x2* is %.2f’ \ % (prob.value, x1.value, x2.value)
# (d)
prob.objective = cvx.Minimize(cvx.max_elemwise(x1, x2)) prob.solve()
# (e)
prob.objective = cvx.Minimize(cvx.square(x1) + 9*cvx.square(x2)) prob.solve()
print ’With obj. x1^2+9x2^2, p* is %.2f, x1* is %.2f, x2* is %.2f’ \ % (prob.value, x1.value, x2.value)
# exercise 4.1 using CVX using Convex
# set up a vector to store optimal values of problems optimal_values=zeros(5); # part a x = Variable(2) obj = x[1] + x[2] constr = [2*x[1]+x[2] >= 1, x[1]+3*x[2] >= 1, x >= 0] p = minimize(obj, constr) solve!(p) optimal_values[1]=p.optval; # part b x = Variable(2) obj = -sum(x) constr = [2*x[1]+x[2] >= 1, x[1]+3*x[2] >= 1, x >= 0] p = minimize(obj, constr) solve!(p) optimal_values[2]=p.optval; # part c x = Variable(2) obj = x[1] constr = [2*x[1]+x[2] >= 1, x[1]+3*x[2] >= 1, x >= 0] p = minimize(obj, constr) solve!(p) optimal_values[3]=p.optval; # part d x = Variable(2) obj = maximum(x) constr = [2*x[1]+x[2] >= 1, x[1]+3*x[2] >= 1, x >= 0] p = minimize(obj, constr) solve!(p) optimal_values[4]=p.optval; # part e
x = Variable(2)
obj = square(x[1]) + 9*square(x[2])
constr = [2*x[1]+x[2] >= 1, x[1]+3*x[2] >= 1, x >= 0] p = minimize(obj, constr)
solve!(p)
optimal_values[5]=p.optval;
3.3 Reformulating constraints in CVX*. Each of the following CVX code fragments describes a convex constraint on the scalar variables x, y, and z, but violates the CVX rule set, and so is invalid. Briefly explain why each fragment is invalid. Then, rewrite each one in an equivalent form that conforms to the CVX rule set. In your reformulations, you can use linear equality and inequality constraints, and inequalities constructed using CVX functions. You can also introduce additional variables, or use LMIs. Be sure to explain (briefly) why your reformulation is equivalent to the original constraint, if it is not obvious.
Check your reformulations by creating a small problem that includes these constraints, and solving it using CVX. Your test problem doesn’t have to be feasible; it’s enough to verify that CVX processes your constraints without error.
Remark. This looks like a problem about ‘how to use CVX software’, or ‘tricks for using CVX’. But it really checks whether you understand the various composition rules, convex analysis, and constraint reformulation rules.
(a) norm([x + 2*y, x - y]) == 0 (b) square(square(x + y)) <= x - y (c) 1/x + 1/y <= 1; x >= 0; y >= 0 (d) norm([max(x,1), max(y,2)]) <= 3*x + y (e) x*y >= 1; x >= 0; y >= 0 (f) (x + y)^2/sqrt(y) <= x - y + 5 (g) x^3 + y^3 <= 1; x >= 0; y >= 0 (h) x + z <= 1 + sqrt(x*y - z^2); x >= 0; y >= 0 Solution.
(a) The lefthand side is correctly identified as convex, but equality constraints are only valid with affine left and right hand sides. Since the norm of a vector is zero if and only if the vector is zero, we can express the constraint as x + 2*y == 0; x - y == 0, or simply x == 0; y == 0. (b) The problem is that square() can only accept affine arguments, because it is convex, but not
increasing. To correct this use square_pos() instead: square_pos(square(x + y)) <= x - y
We can also reformulate this constraint by introducing an additional variable. variable t
Note that, in general, decomposing the objective by introducing new variables doesn’t need to work. It works in this case because the outer square function is convex and monotonic over R+.
Alternatively, we can rewrite the constraint as (x + y)^4 <= x - y
(c) 1/x isn’t convex, unless you restrict the domain to R++. We can write this one as inv_pos(x) + inv_pos(y) <= 1.
The inv_pos function has domain R++ so the constraints x > 0, y > 0 are (implicitly)
in-cluded.
(d) The problem is that norm() can only accept affine argument since it is convex but not increas-ing. One way to correct this is to introduce new variables u and v:
norm([u, v]) <= 3*x + y max(x,1) <= u
max(y,2) <= v
Decomposing the objective by introducing new variables works here because norm is convex and monotonic over R2+, and in particular over [1,∞) × [2, ∞).
(e) xy isn’t concave, so this isn’t going to work as stated. But we can express the constraint as x >= inv_pos(y). (You can switch around x and y here.) Another solution is to write the constraint as geo_mean([x, y]) >= 1. We can also give an LMI representation:
[x 1; 1 y] == semidefinite(2)
(f) This fails when we attempt to divide a convex function by a concave one. We can write this as
quad_over_lin(x + y, sqrt(y)) <= x - y + 5
This works because quad_over_lin is monotone decreasing in the second argument, so it can accept a concave function here, and sqrt is concave.
(g) The function x3+ y3 is convex for x ≥ 0, y ≥ 0. But x3 isn’t convex for x < 0, so CVX is
going to reject this statement. One way to rewrite this constraint is
quad_pos_over_lin(square(x),x) + quad_pos_over_lin(square(y),y) <= 1
This works because quad_pos_over_lin is convex and increasing in its first argument, hence accepts a convex function in its first argument. (The function quad_over_lin, however, is not increasing in its first argument, and so won’t work.)
Alternatively, and more simply, we can rewrite the constraint as pow_pos(x,3) + pow_pos(y,3) <= 1
(h) The problem here is that xy isn’t concave, which causes CVX to reject the statement. To correct this, notice that
q
xy− z2 =qy(x− z2/y),
so we can reformulate the constraint as
This works, since geo_mean is concave and nondecreasing in each argument. It therefore accepts a concave function in its first argument.
We can check our reformulations by writing the following feasibility problem in CVX (which is obviously infeasible) cvx_begin variables x y u v z x == 0; y == 0; (x + y)^4 <= x - y; inv_pos(x) + inv_pos(y) <= 1; norm([u; v]) <= 3*x + y; max(x,1) <= u; max(y,2) <= v; x >= inv_pos(y); x >= 0; y >= 0; quad_over_lin(x + y, sqrt(y)) <= x - y + 5; pow_pos(x,3) + pow_pos(y,3) <= 1; x+z <= 1+geo_mean([x-quad_over_lin(z,y), y]) cvx_end
3.4 Optimal activity levels. Solve the optimal activity level problem described in exercise 4.17 in Convex Optimization, for the instance with problem data
A = 1 2 0 1 0 0 3 1 0 3 1 1 2 1 2 5 1 0 3 2 , cmax= 100 100 100 100 100 , p = 3 2 7 6 , pdisc = 2 1 4 2 , q = 4 10 5 10 .
You can do this by forming the LP you found in your solution of exercise 4.17, or more directly, using CVX*. Give the optimal activity levels, the revenue generated by each one, and the total revenue generated by the optimal solution. Also, give the average price per unit for each activity level, i.e., the ratio of the revenue associated with an activity, to the activity level. (These numbers should be between the basic and discounted prices for each activity.) Give a very brief story explaining, or at least commenting on, the solution you find.
Solution. For this part, we write the problem in a form close to its original statement, and let CVX* do the work of reformulating it as an LP.
The following CVX code implements this:
A=[ 1 2 0 1; 0 0 3 1; 0 3 1 1;
1 0 3 2]; cmax=[100;100;100;100;100]; p=[3;2;7;6]; pdisc=[2;1;4;2]; q=[4; 10 ;5; 10]; cvx_begin variable x(4) maximize( sum(min(p.*x,p.*q+pdisc.*(x-q))) ) subject to x >= 0; A*x <= cmax cvx_end x r=min(p.*x,p.*q+pdisc.*(x-q)) totr=sum(r) avgPrice=r./x import cvxpy as cvx import numpy as np A = np.matrix(’1 2 0 1; \ 0 0 3 1; \ 0 3 1 1; \ 2 1 2 5; \ 1 0 3 2’) cmax = np.matrix(’100; 100; 100; 100; 100’) p = np.matrix(’3; 2; 7; 6’) pdisc = np.matrix(’2; 1; 4; 2’) q = np.matrix(’4; 10; 5; 10’) x = cvx.Variable(4) t1 = cvx.mul_elemwise(p, x) t2 = cvx.mul_elemwise(p, q) + cvx.mul_elemwise(pdisc, x-q) obj = cvx.Maximize(cvx.sum_entries(cvx.min_elemwise(t1, t2)))
cons = [x >= 0, A*x <= cmax]
prob = cvx.Problem(obj, cons) prob.solve()
totr = sum(r) avgPrice = r / x.value print x.value print r print totr print avgPrice using Convex A = [ 1 2 0 1; 0 0 3 1; 0 3 1 1; 2 1 2 5; 1 0 3 2]; cmax = [100;100;100;100;100]; p = [3;2;7;6]; pdisc = [2;1;4;2]; q = [4; 10 ;5; 10]; x = Variable(4); r = min(p.*x, p.*q + pdisc.*(x - q)); totr = sum(r);
constraints = [x >= 0, A*x <= cmax]; p = maximize(totr, constraints); solve!(p); avgPrice = evaluate(r)./evaluate(x); println(x.value) println(evaluate(r)) println(evaluate(totr)) println(avgPrice)
The result of the code is
x = 4.0000 22.5000 31.0000 1.5000 r =
