Linear Programming is a set of techniques for solving linear optimisation problems.
We can define an optimisation problem as finding the best assignment of a set of decision variables subject to:
• an optimisation sense (minimisation or maximisation);
• an objective function, which assigns a cost vector to the variables;
• a set of constraints on the variables; and,
• a domain for the variables.
Linear Programming is defined as having: a set of continuous variables, linear inequalities and a linear objective function. The combination of inequalities describes a feasible region2 which forms a convex polytope.
Definition 1.1. A polytope is a set defined as the intersection of finitely many half-spaces, each of which is defined by an inequality.
Definition 1.2. An objective function is a real-valued function defined on a polytope.
The goal in LP is to find the point3 in the polytope where this function has the smallest (or largest) value. For example, the simplex algorithm operates on linear programs by constructing a feasible solution at a vertex of the polytope and then walking along a path on the edges of the polytope to vertices with non-decreasing values of the objective function until an optimum is reached.
1.1.1 Feasible set
We define the feasible set4 X of a linear program as the set of n variables x respecting a set of inequalities defined by a matrix A of size m×n and a right-hand side vector b of size m.
X= {x∈Rn | Ax≥b}
We will only consider problems with a non-empty feasible region, thus the convex hull of X, conv(X), is a non-empty polytope. From now on, we will assume the problems to be well-defined and will thus omit the dimension of the different entities.
1.1.2 Linear programs
A linear program is a mathematical representation of an optimisation problem which consists in finding the optimal assignment of variables within the feasible set given a cost vector c and a direction.
Definition 1.3. The standard form of a linear program is: a minimisation problem with greater-or-equal constraints and positive variables.
With this definition, we write a linear program in standard form as follows:
min cTx (Std)
s.t. Ax≥ b (1.1a)
x≥0
2Otherwise, it is easy to prove that the problem is infeasible.
3We can sometimes have more than one optimal solution, in which case we call the problem “de-generate.”
4Sometimes called solution set.
This property follows from using the following procedure:
1. A maximisation can be turned into a minimisation by multiplying the objective coefficients’ by−1: max cTx ⇐⇒ min−cTx
2. Similarly, any lesser-or-equal constraint can be changed into a greater-or-equal:
Ax≤b ⇐⇒ −Ax≥ −b
(a) In case of an equality constraint, it can be replaced by two constraints: one greater- and one lesser-or-equal-than.
3. Negative variables can be replaced by their opposite.
(a) In case a variable is unrestricted, it can be replaced by two non-negative variables: xi =ui−vi.
Because of this property, we will no longer ensure that linear programs are writ-ten in standard form but use the most intuitive notation.
1.1.2.1 Toy problem
Take the following linear program, and its associated graph presented in Figure 1.1.
max x1+x2 (LP)
s.t. x1+2x2 ≤4 (1.2a)
4x1+2x2 ≤12 (1.2b)
−x1+x2 ≤1 (1.2c)
x1, x2 ≥0
The solving process is illustrated in Figure 1.2 and described in full in Ap-pendix C. The simplex method starts at an arbitrary node: (0, 0) here; then tries to find a direction where the objective function increases along an edge of the polytope.
1.1.3 Duality
To select which constraint vector to follow next, the simplex uses linear duality. Du-ality is another way to look at a linear program: minimising an objective function is equivalent to maximising resource utilisation.
In this case, what we call “resource” is the amount by which the objective function would increase if we modified the right-hand side of a constraint. This is called the shadow price of a constraint. The foundation of linear duality is that for every primal linear program, there exists a unique dual linear program which operates on the same data and has the same optimal value.
1 2 3 4 with (LP), the blue area is the feasible
set. Figure 1.2: Example of the simplex
method starting at (0, 0)and walking until the optimum (red dot) is found;
the dashed line and arrow is the ob-jective’s direction.
First, let us define how to obtain the dual problem given a linear program. Given a problem in standard form, such as (Std), we define α the dual variable (shadow price) associated with its constraint set (1.1a); then, its dual is defined as:
max bTα (Dual)
s.t. ATα≤ c (1.3a)
α≥0
Formally, the dual is the negative transpose of the primal. This can be observed in: the optimisation sense is reversed, the constraints set’s right-hand side vector is used as objective, the objective vector as right-hand side, and the constraint matrix transposed.
Table 1.1 summarises the relationship between constraints and variables when dualising a linear program.
Constraint Variable
≤ ≥0
≥ ≤0
= ∈R
Table 1.1: Relation between constraints and variables when dualising.
Theorem 1.1(Weak Duality). If ¯x is a feasible solution for the primal minimisation linear program and ¯α is a feasible solution for the dual maximisation linear program, then:
∑
ibi¯αi ≤
∑
j
cj¯xj,
where cj and bi are the coefficients of the respective objective functions.
Proof. The weak duality property follows immediately from the respective feasibility of the two solutions:
1. for the primal, we have: ¯x≥0 and A ¯x≥ b; and, 2. for the dual, we have: ¯α ≥0 and AT¯α ≤c.
Hence, multiplying each primal constraint by its shadow price and adding – conversely for the dual – yields:
∑
i∑
Because the left-hand side of both inequalities is the same, by combining them we have the desired result.
Weak duality means that solving the dual problem always gives a lower bound on the objective of the primal problem. We can derive the optimality and unboundedness properties from this result:
Lemma 1.2(Certificate of Optimality). If ¯x is a feasible solution to the primal problem and
¯α is a feasible solution to the dual problem, and, further:
∑
ibi¯αi =
∑
j
cj¯xj,
then ¯x is an optimal solution to the primal problem and ¯α is an optimal solution to the dual problem.
Lemma 1.3 (Unboundedness and Feasibility). If the primal (dual) problem has an un-bounded solution, then the dual (primal) problem is infeasible.
Given these properties, we can now proceed to the main result of linear duality:
Theorem 1.4 (Strong Duality). If the primal (dual) problem has a finite optimal solution, then so does the dual (primal) problem, and these two values are equal.
The proof is left out as it is quite lengthy, but it can be found in any good book about linear programming [79]. The intuition is: if the primal solution is optimal, some constraints are active5; therefore, their dual variables are zero; thus the
max-5Their left-hand side is equal to their right-hand side.
imum amount of resource available in the dual has to be equal to the value of the primal, otherwise there would be a primal constraint with slack.
In other words, if there is an optimal solution there is no optimality gap. This result is at the centre of a number of optimisation algorithms. For example, if a problem has a large number of constraints but few variables, it may be easier to solve its dual.
1.1.3.2 Dual of the toy problem
Recall the previous example (LP). If we associate α{a,b,c}with its constraints Eqs. (1.2a)–(1.2c) the we have the following dual:
min 4αa+12αb+αc (Toy Dual)
s.t. αa+4αb−αc≥1 (1.6a)
2αa+2αb+αc ≥1 (1.6b)
αa, αb, αc ≥0