Consort: Node-constrained Opportunistic Routing in Wireless Mesh Networks

Consort: Node-constrained Opportunistic Routing in Wireless Mesh Networks

Xi Fang, Dejun Yang, and Guoliang Xue

School of Computing, Informatics, and Decision Systems Engineering, Arizona State University

Abstract— Opportunistic routing is proposed to improve the performance of wireless networks by exploiting the broadcast nature and spatial diversity of the wireless medium. In this paper, we study the problems of how to choose opportunistic route for each user to optimize the total utility or profit of multiple simultaneous users in a wireless mesh network (WMN) subject to node constraints. We formulate these two problems as two convex programming systems. By combining primal- dual and subgradient methods, we present a distributed iterative algorithm Consort (node-CONStrained Opportunistic RouTing).

In each iteration, Consort updates Lagrange multipliers in a distributed manner according to the user and node behaviors obtained in the previous iteration, and then each user and each node individually adjusts its own behavior based on the updated Lagrange multipliers. We prove the convergence of this iterative algorithm, and provide bounds on the amount of feasibility violation and the gap between our solution and the optimal solution in each iteration.

1. I NTRODUCTION

Traditional wireless routing often follows the design methodology for wired networks by abstracting the wireless links as wired links to look for the shortest delay or least cost path(s) for a user between a pair of source and destination nodes [12]. However for unreliable wireless networks, the for- warding capacity of intermediate nodes, which overhear packet transmissions, can be explored to improve the performance of WMNs. This observation motivates the emergence of a novel technique known as opportunistic routing, which allows any node overhearing a packet to participate in forwarding it instead of deterministically choosing the next hop before trans- mitting a packet [4][6][11][21]. [4] designed ExOR, an oppor- tunistic routing protocol for WMNs. [6] introduced MORE by combining opportunistic routing and network coding together.

Each user obtains an amount of utility if a certain infor- mation rate is allocated to it. Additionally, since intermediate forwarding nodes provide transmission service for the user, they may charge the user a service fee (i.e. cost). Hence in the economic perspective, each user tries to maximize its utility or profit (i.e. utility minus cost) by choosing an appropriate user behavior (i.e. an opportunistic route).

However, when there exist multiple simultaneous users in a WMN, we will ask how to optimally choose an opportunistic route for each user to maximize the total user utility or profit by solving the resource competition among users, since individual user behaviors usually may not lead to a global optimum. In order to seek for optimized bandwidth usage and congestion avoidance, [33] introduced a game theory

Xi Fang, Dejun Yang, and Guoliang Xue are affiliated with Arizona State University, Tempe, AZ 85281. Email: {xi.fang, dejun.yang, xue}@asu.edu.

This research was supported in part by NSF grants 0830739 and 0905603, and ARO grant W911NF-09-1-0467. The information reported here does not reflect the position or the policy of the federal government.

framework Dice to find a Nash bargaining point [14] of user rates in wireless multipath network coding. However, [33]

mainly focuses on the efficiency-fairness tradeoff of user rates.

Furthermore, nodes, another kind of active network entities, also have their own node constraints, and thus have their own node behaviors. This paper considers the following two node constraints, which characterize a node indi- vidual requirement and a social requirement, respectively.

The first one is that each node may have its own energy consumption constraint, lifetime concern or computation and transmission capacity constraints. Thus it needs to consider the maximum load it is willing to carry, which we call the node max load constraint. The second node constraint is the node load balance, an important concern in network design [15][24][26][27][34]. Without considering node load balance, the philosophy that the optimal route is always chosen could result in congestion on the center of a network or hotspots which drain the energy of the nodes in congested regions or hotspots much faster [20][24]. Security may also be an issue without considering node load balance [24]. For example, if a large number of messages go through a small number of nodes, then jamming can be a vicious attack. In contrast, it is much more expensive and less effective to jam a large number of nodes. Thus, a node may set a maximum allowed load difference compared with other nodes (such as k-hop neighbors), which we call the node load balance constraint. This constraint can ensure that a node will not carry too much more load than other nodes, and thus can smooth the loads among different nodes. In brief, the node max load constraint reflects a node’s own load requirement, and the node load balance constraint shapes the load relationships among different nodes (i.e. social load patterns). Obviously, node constraints and the associated behaviors will affect user utilities and profits, and make user routing selections more difficult.

In this paper, we study the problem of how to choose op- portunistic route for each user to optimize the total utility or profit of multiple simultaneous users in a WMN subject to node constraints. We call these two problems node- constrained user utility optimization problem (NCUUOP) and node-constrained user profit optimization problem (NCUPOP).

In the literature there exist some works which study routing problems taking into account the node load concern. [31]

considered the routing problem with the goal of maximizing the life time of the wireless network. [26] showed that multi- path routing provides better congestion and traffic balancing.

Further work [15] showed that multi-path routing can balance load only if a very large number of paths are used. [27]

showed that an optimum routing scheme based on shortest

paths can be expressed in terms of geometric optics and

computed by linear programming. [34] tried to solve the energy-efficiency issues by balancing the load reactively. [8]

proposed a receiver-oriented load-balancing opportunistic rout- ing protocol for wireless sensor networks. [9] proposed a load-aware routing scheme using dual decomposition. [22]

tried to maximize user utility using multi-path routing in communication networks. The difference between our work and the above line of research is that our work considers given node load requirements as constraints, and subject to them we allocate resource to optimize the total utility or profit of multiple simultaneous users existing in a WMN by using opportunistic routing. Furthermore, our algorithm is distributed, and thus enhances network scalability and self-organization.

Our contribution is two-fold:

1) We mathematically formulate NCUUOP and NCUPOP as two convex programming systems. To the best of our knowledge, this paper represents the first attempt to optimize opportunistic routing for multiple simultaneous users considering node load constraints.

2) By combining primal-dual methods [5] and subgradient methods [30], we present a unified distributed iterative algorithmic framework Consort for both problems. In each iteration, Consort updates Lagrange multipliers in a distributed manner according to the user/node behaviors and the Lagrange multipliers obtained in the previous iteration, and then each user and each node individually adjusts its own behavior based on the updated Lagrange multipliers. We prove its convergence, and also provide bounds on the amount of feasibility violation and the gap between our solution and the optimal one in each iteration. The feature of each user and each node adjust- ing their behaviors in a distributed manner is of great importance for network scalability and self-organization.

Extensive simulations show that Consort can provide higher total utility and profit of users than MORE and Dice when no node constraint is considered, and that Consort can provide a much higher fairness level of node loads and a smaller node max load violation ratio when node constraints are considered.

The rest of this paper is organized as follows. Section 2 reviews the opportunistic routing protocols we consider in this paper. Section 3 describes the system model. Sections 4 and 5 formulate the problems we are studying, and the distributed algorithm for these two problems is presented in Section 6.

Section 7 discusses the practical issues and extensions of our results. Section 8 reveals our simulation results. We conclude this paper in Section 9.

2. P RELIMINARIES

Before formulating the problem to study, we first review the opportunistic routing protocols we consider. A basic op- portunistic routing protocol (e.g. MORE [6]) works as follows.

Source node: The source node divides its traffic into a number of batches with M packets in it, and continuously generates coded packet streams from each batch using a random linear network coding scheme. That is to say a coded packet is m ^ _j =



i c _ji m _i , where c _ji is a random coefficient, and m _i is a native packet from the current batch. {c j1 , ..., c _ji , ..., c _jM } is called the code vector of packet m ^ _j . The source node keeps sending coded packets in a batch until an uncoded ACK for this batch is sent back to the source.

Intermediate node: When an intermediate node hears a packet, it checks whether it is in this packet’s forwarder list (a list including nodes that could participate in forwarding the packet), and whether the packet contains new information (i.e. an innovating packet) using simple algebra, such as Gaussian Elimination. A packet is innovative if it is linearly independent from the packets the node has previously received from this batch. This node then creates a random linear combination of the coded packets it has from the same batch and broadcasts it. Suppose that an intermediate node has coded packets of the form m ^ _j = 

i c _ji m _i . It creates more coded packets by linearly combining these coded packets as follows:

m ^ = 

j a _j m ^ _j , where a _j ’s are random numbers. Note that m ^ is also a linear combination of the native packets (i.e.

m ^ = 

i ( 

j a _j c _ji )m _i ).

Destination node: The destination keeps all the received inno- vative packets until M innovative packets are received. Then it decodes the original whole batch using matrix inversion and sends an acknowledgement back to the source node using a traditional routing protocol.

3. S YSTEM M ODEL

In this paper, we consider static or slowly changing dynamic WMNs, such as Roofnet [1] and community wireless networks [2][28]. There exist K simultaneous users, where each user maintains a session from a source node to a destination node and uses intermediate nodes to relay the packets. Our work is particularly suitable for the applications with long user sessions, such as reliable large file transfer. We will discuss the case of shorter sessions in Section 7. We need the following sub-models to formulate NCUUOP and NCUPOP.

A. Opportunistic multipath routing network sub-model We mainly use the model introduced in [33] to formulate the basic properties of opportunistic routing. This model has been shown to be sufficiently effective and tractable for network analysis. We model a WMN as a directed graph G = (V, E), where V is the set of vertices, and E is the set of edges. We use the following terms interchangeably: edge and link, vertex and node. Unlike the traditional unit disk model, the transmission range is defined as the distance where the reception probability falls below a small threshold. If v is in u’s transmission range, there is a directed edge (u, v) in G. As in [6], a distributed node pre-selection procedure is performed to add the nodes into the intermediate forwarder list for a user, such that each forwarder is closer (in ETX metric [10]) to the destination than its predecessors. We use G(V _k , E _k ) to denote the resulting topology for user k, where V _k and E _k are the set of nodes and the set of directed links involved in the session of user k.

1) Multipath flow conservation constraint: Considering that flow conservation holds for each node, we must have



(u,v)∈E k

r _k (u, v) − 

(w,u)∈E k

r _k (w, u) = h _k (u), ∀k ∈ [1, K],

where h _k (u) is equal to λ _k if u = s _k , −λ k if u = d _k , and 0 otherwise. r _k (u, v) denotes the information flow rate of user k on link (u, v). λ _k , s _k and d _k denote the information rate, the source and the destination of user k. Mathematically, the information flow rate vector − → r _k represents the behavior of user k (i.e. the opportunistic route of user k).

2) MAC broadcasting rate constraint: We consider the collision free broadcast MAC formulated in [33]: a broadcast transmission from node u is collision free if and only if all other transmitters are outside the range of any downstream receiver of node u. Since the transmission range is defined as the distance where the reception probability falls below a small threshold, it is fair to assume that the interference range equals to the transmission range [33]. We use R(u) to denote the set of nodes whose transmission range (interference range) node u is located in.

As in [33], we consider that a slotted scheduling-based collision-free broadcast MAC is being used. Note that although we design Consort based on this collision free broadcast MAC model, Section 7 will discuss how to integrate Consort to support 802.11 MAC. Let B _k ^(t) (u) denote a binary decision variable indicating whether node u is transmitting user k’s data in slot t. Thus, a necessary and sufficient condition is:



k∈[1,K]

B _k ^(t) (u) + 

k∈[1,K]



v∈R(u)

B _k ^(t) (v) ≤ 1, ∀u ∈ V k \ s k . (3.1)

(3.1) respects the fact that in each time slot, the receive antenna of any node u allows the broadcast transmission from at most one transmitter within its range (including itself). Note that for user k the source node s _k is excluded, since s _k does not need to receive information from other nodes for this user.

Considering the schedule length of T , based on (3.1) we have C

T



t∈[1,T ]



k∈[1,K]

B _k ^(t) (u) + C T



t∈[1,T ]



k∈[1,K]



v∈R(u)

B _k ^(t) (v) ≤ C,

∀u ∈ V _k \ s _k , (3.2)

where C is the MAC layer capacity, which is the maximum broadcast rate of a node when no interferer presents. Since the average broadcast rate of node u for user k (i.e. node u’s behavior for user k) can be computed as b _k (u) = lim _{T →∞} ^C

T



t∈[1,T ] B _k ^(t) (u), we must have



k∈[1,K]

b _k (u) + 

k∈[1,K]



v∈R(u)

b _k (v) ≤ C, ∀u ∈ V k \ s k . (3.3)

Note that (3.3) is necessary but not necessarily sufficient, as we transformed an integer variable B _k ^(t) (u) into a continuous one b _k (u) by averaging.

3) Coding constraint: The coding condition should satisfy the following straightforward network coding model.

b _k (u) · p(u, v) ≥ r k (u, v), ∀(u, v) ∈ E k , (3.4) where p(u, v) is the packet delivery ratio of link (u, v).

As discussed in [33], this is not a tight bound. However this involves approximations to the behavior of an actual WMN, and includes all the tractable information that users can use to induce a better payoff. [23] made an exact characterization

of the coding model. However with an exponential number of constraints, it makes the problem intractable.

B. Node constraint sub-model

1) Node max load constraint: Since each active node u may have its own energy consumption constraint, lifetime concern or computation and transmission capacity constraints, it needs to consider the maximum average broadcast rate ϕ(u) it is willing to provide. Thus each active node sets an upper bound on its average broadcast rate:

b(u) = 

k∈[1,K]

b _k (u) ≤ ϕ(u), ϕ(u) > 0. (3.5)

Consider, for instance, nodes with the concern of lifetime. As discussed in [32], in most types of wireless network nodes, wireless transceivers dominate the energy consumption. Each active node u can estimate ϕ(u) as follows: ϕ(u) = _{T (u)ε(u)} ^E(u) , where E(u) is its remaining energy, T (u) is its expected lifetime, and ε(u) is its average energy consumption of trans- mitting one bit.

2) Node load balance constraint: We define a load balance area (i.e. a set of nodes) for each node. Let v ∈ A(u) represent that v is in u’s load balance area. That is to say,

|b(u) − b(v)| ≤ θ(u, v), θ(u, v) > 0, ∀v ∈ A(u), (3.6) where θ(u, v) is a parameter set by node u. We introduce A(u) and θ(u, v) to help u have a controllable balanced load with other nodes. In this paper, we consider a symmetric definition (i.e. v ∈ A(u) ⇐⇒ u ∈ A(v), and θ(u, v) = θ(v, u)).

In practice, the required load balance constraint could be asymmetric. However, there would be no big difference for our mathematical analysis and algorithm (see Sections 6), and we thus focus on this simpler definition which makes our expressions clearer.

C. User utility and profit sub-model

1) User utility function: We use the following common utility function model. If an information rate λ _k is allocated to a user k, then user k has a utility of U _k (λ _k ). U _k (·) is an increasing, strictly concave and continually differentiable function, which is a common assumption of utility func- tions [19][29].

2) User profit function: Since intermediate forwarders pro- vide transmission service for a user, each forwarder charges this user a service fee which is proportional to the service rate (i.e. the broadcast rate) for this user. We use σ _k (u) to denote the service price of node u for user k. A user k therefore should pay σ _k (u)b _k (u) (σ _k (u) ≥ 0) for node u, and the profit function for user k is thus defined as

P _k (λ _k , − →

b _k ) = U _k (λ _k ) − 

u∈V k \d k

σ _k (u)b _k (u). (3.7)

4. N ODE - CONSTRAINED U SER U TILITY O PTIMIZATION

A. Problem formulation

We formulate NCUUOP as the following system:

System1(− → r , − → b ) :

Maximize 

k∈[1,K]

U _k (λ _k ) (4.1)

s.t. 

(u,v)∈E k

r _k (u, v) − 

(w,u)∈E k

r _k (w, u)=h _k (u), u∈V k , k∈[1, K], (4.2)



k∈[1,K]

b _k (u) + 

k∈[1,K]



v∈R(u)

b _k (v) ≤ C, ∀u ∈ V k \ s k , (4.3) b _k (u) · p(u, v) ≥ r  _k (u, v), ∀(u, v) ∈ E _k , k ∈ [1, K], (4.4)

k∈[1,K]

b _k (u) ≤ ϕ(u), ∀u ∈ V k \ d k , (4.5)

| 

k∈[1,K]

b _k (u ₁ ) − 

k∈[1,K]

b _k (u ₂ )| ≤ θ(u ₁ , u ₂ ),

∀u 1 = u 2 , u ₁ ∈ V k \ d k , u ₂ ∈ V k \ d k , u ₂ ∈ A(u 1 ), (4.6) over : r _k (u, v) ∈ [0, C], ∀(u, v) ∈ E k , k ∈ [1, K],

b _k (u) ∈ [0, C], ∀u ∈ V k \ d k , k ∈ [1, K].

B. Lagrangian analysis

Due to our symmetric definition of load balance area, the absolute value sign in (4.6) can be omitted. For convenience, we set b _k (u) = 0, if u / ∈ V k \ d k . This can be considered as a supplement constraint for System 1. We thus compute the Lagrangian of System 1 subject to constraint (4.2) as follows:

L(− → r , − → b , − → α , − →

β , − → μ , − → ω ) = − 

k∈[1,K]

U _k (λ _k )

+ 

u∈V

α(u)  

k∈[1,K],u=s k

b _k (u) + 

k∈[1,K]



v∈R(u),u=s k

b _k (v) − C 

+ 

k∈[1,K]



(u,v)∈E k

β _k (u, v) 

r _k (u, v) − b k (u)p(u, v) 

+ 

u∈V

μ(u)  

k∈[1,K]

b _k (u) − ϕ(u) 

+ 

u 1 ,u 2 ∈V :u 2 ∈A(u 1 )

ω(u ₁ , u ₂ ) 

k∈[1,K]

 b _k (u ₁ ) − b k (u ₂ ) 

− θ(u ₁ , u ₂ )  . (4.7)

After some mathematical manipulations, (4.7) leads to the following:

L(− → r , − → b , − → α , − →

β , − → μ , − → ω )

=− 

k∈[1,K]

U _k (λ _k ) + 

k∈[1,K]



(u,v)∈E k

β _k (u, v)r _k (u, v)

+ 

k∈[1,K]



u∈V k \s k

α(u)b _k (u) + 

k∈[1,K]



u∈V k



v∈R(u),v=s k

α(v)b _k (u)

− 

k∈[1,K]



(u,v)∈E k

β _k (u, v)b _k (u)p(u, v) + 

u∈V

μ(u) 

k∈[1,K]

b _k (u)

+ 

u 1 ,u 2 ∈V :u 2 ∈A(u 1 )

ω(u ₁ , u ₂ ) 

k∈[1,K]

 b _k (u ₁ ) − b k (u ₂ ) 

−C 

u∈V

α(u)

− θ(u 1 , u ₂ ) 

u 1 ,u 2 ∈V :u 2 ∈A(u 1 )

ω(u ₁ , u ₂ ) − 

u∈V

μ(u)ϕ(u). (4.8)

(4.8) implies that the minimization operation of L(− → r , − →

b , − → α , − →

β , − → μ , − → ω ) over (− → r , − →

b ) can be decomposed as

each user’s behavior and each node’s behavior for each user.

1) User k’s behavior:

Minimize −U k (λ _k ) + 

(u,v)∈E k

β _k (u, v)r _k (u, v), (4.9)

s.t. 

(u,v)∈E k

r _k (u, v) − 

(w,u)∈E k

r _k (w, u) = h _k (u), u ∈ V k

over : r _k (u, v) ∈ [0, C], ∀(u, v) ∈ E k . 2) Nodes u’s behavior for user k:

Minimize b _k (u) 

α(u) _u=s

k + 

v∈R(u),v=s k

α(v) − 

(u,v)∈E k

β _k (u, v)p(u, v)

+ 

v∈A(u)

ω (u, v) − 

v∈A(u)

ω(v, u) + μ(u) 

, (4.10) over: b _k (u) ∈ [0, C], u ∈ V k \ d k .

Combining this decomposition (4.9) and (4.10) with the subgradient methods [25][30], a distributed algorithm for solving System 1 is hence possible. This algorithm will be presented in Section 6.

5. N ODE - CONSTRAINED U SER P ROFIT O PTIMIZATION

A. Problem formulation

We formulate NCUPOP as the following system:

System2(− → r , − → b ) : Maximize 

k∈[1,K]



U _k (λ _k ) − 

u∈V k \d k

σ _k (u)b _k (u) 

, (5.1)

s.t. 

(u,v)∈E k

r _k (u, v) − 

(w,u)∈E k

r _k (w, u)=h _k (u), u∈V k , k ∈[1, K], (5.2)



k∈[1,K]

b _k (u) + 

k∈[1,K]



v∈R(u)

b _k (v) ≤ C, ∀u ∈ V _k \ s _k , (5.3) b _k (u) · p(u, v) ≥ r  k (u, v), ∀(u, v) ∈ E k , k ∈ [1, K], (5.4)

k∈[1,K]

b _k (u) ≤ ϕ(u), ∀u ∈ V k \ d k , (5.5)

| 

k∈[1,K]

b _k (u ₁ ) − 

k∈[1,K]

b _k (u ₂ )| ≤ θ(u 1 , u ₂ ),

∀u 1 = u 2 , u ₁ ∈ V k \ d k , u ₂ ∈ V k \ d k , u ₂ ∈ A(u 1 ), (5.6) over : r _k (u, v) ∈ [0, C], ∀(u, v) ∈ E k , k ∈ [1, K],

b _k (u) ∈ [0, C], ∀u ∈ V k \ d k , k ∈ [1, K].

B. Lagrangian analysis

As in Section 4.B, we compute the Lagrangian of System 2 subject to constraint (5.2) as follows:

L(− → r , − → b , − → α , − →

β , − → μ , − → ω ) =− 

k∈[1,K]

U _k (λ _k ) + 

k∈[1,K]



u∈V

σ _k (u)b _k (u)

+ 

u∈V

α(u)  

k∈[1,K],u=s k

b _k (u) + 

k∈[1,K]



v∈R(u),u=s k

b _k (v) − C 

+ 

k∈[1,K]



(u,v)∈E k

β _k (u, v) 

r _k (u, v) − b k (u) · p(u, v) 

+ 

u∈V

μ(u)  

k∈[1,K]

b _k (u) − ϕ(u) 

+ 

u 1 ,u 2 ∈V :u 2 ∈A(u 1 )

ω(u ₁ , u ₂ )  

k∈[1,K]

 b _k (u ₁ )−b k (u ₂ ) 

−θ(u ₁ , u ₂ ) 

. (5.7)

As in Section 4, after some mathematical manipu- lations, we can decompose the problem of minimizing L(− → r , − →

b , − → α , − →

β , − → μ , − → ω ) over (− → r , − →

b ) into each user’s behav- ior and each node’s behavior.

1) User k’s behavior:

Minimize −U k (λ _k ) + 

(u,v)∈E k

β _k (u, v)r _k (u, v), (5.8)

s.t. 

(u,v)∈E k

r _k (u, v)− 

(w,u)∈E k

r _k (w, u) = h _k (u), u ∈ V k

over :r _k (u, v) ∈ [0, C], ∀(u, v) ∈ E k . 2) Node u’s behavior for user k:

Minimize b _k (u) 

α(u) _u=s

k + 

v∈R(u),v=s k

α(v)− 

(u,v)∈E k

β _k (u, v)p(u, v)

+ 

v∈A(u)

ω (u, v) − 

v∈A(u)

ω(v, u) + μ(u) + σ _k (u)  , (5.9) over: b _k (u) ∈ [0, C], u ∈ V k \ d k .

As discussed in Section 4, a distributed iterative algorithm (shown in Section 6) for solving System 2 is also possible.

6. D ISTRIBUTED A LGORITHMIC F RAMEWORK : C ONSORT

In this section, we present a distributed iterative algorithmic framework, called Consort, to solve both Systems 1 and 2.

Consort can be viewed as a variant of subgradient methods [30]. The theory underlying Consort is the approximate dual subgradient methods (ADSM) introduced in [25]. ADSM is a powerful tool to design decentralized resource allocation algo- rithms for networking applications, especially combined with the decomposition mechanism. However, when the Lagrangian or some decomposed part of the Lagrangian is linear, oscil- lations may occur in ADSM iterations (see Section 6.A). We thus design a modified approximate dual subgradient method (MADSM) to deal with this issue at the cost of introducing a controllable inaccuracy (initially inspired by the regulation term technique in [3][22][33]). Note that MADSM is a generic framework. Thus, essentially Consort is an application of MADSM for solving Systems 1 and 2. We will also see that ADSM is a special case of our MADSM.

In the rest of this section, we first present our MADSM and its properties, and then use this framework to design Consort.

A. Modified approximate dual subgradient method The primal problem is the following:

Minimize f (− → x ), s.t. g(− → x ) 0, over: − → x ∈ − → X , (6.1) where f (·) : R ^N −→ R is a convex function, g(·) = (g ₁ (·), g 2 (·), ..., g ρ (·)) ^T and each g _j (·) : R ^N −→ R is a convex function, and − →

X ∈ R ^N is a nonempty compact convex set. Note that (·) ^T denotes the transpose of (·).

The dual problem is the following:

Maximize q( − → δ )= inf

−

→ x ∈ − → X

 L(− → x , − → δ ) 

= inf

−

→ x ∈ − → X

 f (− → x )+− → δ ^T g(− → x ) 

, s.t. − →

δ  0, over: − → δ ∈ R ^ρ ,

where L(− → x , − →

δ ) = f(− → x )+− →

δ ^T g(− → x ) is the Lagrangian of the primal problem, and − →

δ is the vector of Lagrange multipliers.

MADSM sequentially computes − →

δ ⁽¹⁾ , − → x ⁽¹⁾ , − →

δ ⁽²⁾ , − → x ⁽²⁾ , ...

from a starting point { − →

δ ⁽⁰⁾ , − → x ⁽⁰⁾ } ∈ R ^ρ × − →

X as follows:

for i ≥ 0, − →

δ ⁽ⁱ⁺¹⁾ = [− →

δ ⁽ⁱ⁾ + ηg(− → x ⁽ⁱ⁾ )] ⁺ , where − → x ⁽ⁱ⁾ ∈ argmin _{− → x ∈} − →

X

 f (− → x )+(− →

δ ⁽ⁱ⁾ ) ^T g(− → x )+

 N n=1

 _n ||x n −x ⁽ⁱ⁻¹⁾ _n || ²  ,(6.2)

where η is a constant stepsize, − →

δ ⁽ⁱ⁾ and − → x ⁽ⁱ⁾ are the vectors of Lagrange multipliers and solutions generated in the i ^th iteration, respectively, x _n is the n ^th element of − → x , and N is the dimension of − → x . We call  _n ≥ 0 the regulation term of element x _n , which is used to ensure strict convexity. In addition, for a vector − → x ∈ R ^N , − → x ⁺ is the projection of − → x on the nonnegative orthant in R ^N , i.e. − → x ⁺ is the component- wise maximum of the vector − → x and the zero vector: − → x ⁺ = (max{0, x 1 }, ..., max{0, x N }) ^T for − → x = (x ₁ , ..., x _N ) ^T . Note that if all  _n = 0, MADSM reduces to ADSM. If the Lagrangian L(− → x , − →

δ ⁽ⁱ⁾ ) = f(− → x ) + (− →

δ ⁽ⁱ⁾ ) ^T g(− → x ) is a lin- ear function of − → x , obviously the minimization operation of L(− → x , → −

δ ⁽ⁱ⁾ ) over − → x ∈ − →

X by ADSM will lead to oscillations in iterations (i.e. the boundary points are always chosen). When we decompose the Lagrangian, and minimize a part of the Lagrangian, oscillation will also happen for the same reason if this part is linear. The regulation term is hence introduced to ensure strict convexity.

The primal solution is approximated using averaging:

ˆx ⁽ⁱ⁾ = 1 i

 i−1 m=0

−

→ x ^(m) , i ≥ 1. (6.3)

Theorem 6.1 characterizes the properties of MADSM.

Theorem 6.1: If there exists a vector − →

x ^s that satisfies the Slater condition [5] and the norms of the subgradients are bounded by a constant L (i.e.L = max − → x ∈ → −

X ||g(− → x )||), the following properties hold for all i ≥ 1:

1) An upper bound on the amount of constraint viola- tion of the vector ˆ x ⁽ⁱ⁾ is given by ||g(ˆx ⁽ⁱ⁾ ) ⁺ || ≤

B

iη , where the constant B = _ ²  f (− →

x ^s ) − q( − → δ ⁽⁰⁾ ) 

+ max{|| − →

δ ⁽⁰⁾ ||, _ ¹ (f(− → x ^s )−q( − →

δ ⁽⁰⁾ )+ ^ηL ₂ ² +  _N

n=1  _n X _n ² ) + ηL} ( = min j∈[1,ρ] (−g j (− →

x ^s )) and X n = max _{− →}

x ^ ,− → x ^ ∈ − →

X ||x ^ _n − x ^ _n ||). Note that > 0, as − → x ^s satisfies the Slater condition[5].

2) An upper bound on f (ˆ x ⁽ⁱ⁾ ) is given by f(ˆx ⁽ⁱ⁾ ) ≤ f ^∗ +

|| − → δ ⁽⁰⁾ || ²

2iη + ^ηL ₂ ² +  _N

n=1  _n X _n ² , where f ^∗ is the optimal value of the primal objective function.

3) A lower bound on f (ˆ x ⁽ⁱ⁾ ) is given by f(ˆx ⁽ⁱ⁾ )≥f ^∗ − ^B _iη ² .2 The proof is quite involved and we omit the proof due to the space limit. The proof can be found in [13].

B. Consort algorithmic framework

We first briefly describe Consort. Let − → δ ⁽ⁱ⁾ denote the vector of Lagrange multipliers ((− → α ⁽ⁱ⁾ ) ^T , (− →

β ⁽ⁱ⁾ ) ^T , (− → μ ⁽ⁱ⁾ ) ^T , (− → ω ⁽ⁱ⁾ ) ^T ) ^T in the i ^th iteration,

−

→ x ⁽ⁱ⁾ denote the vector of user and node behaviors

((− → r ⁽ⁱ⁾ ) ^T , (− →

b ⁽ⁱ⁾ ) ^T ) ^T in the i ^th iteration, and − → x denote (− → r ^T , − →

b ^T ) ^T . According to MADSM, from a starting point we compute − →

δ ⁽¹⁾ , and based on − →

δ ⁽¹⁾ , − → x ⁽¹⁾ can be computed.

In the second iteration, we compute − →

δ ⁽²⁾ based on − → x ⁽¹⁾ , and based on − →

δ ⁽²⁾ , − → x ⁽²⁾ can be computed. This procedure continues until a stop criteria is reached. Thanks to the decompositions in Sections 4 and 5, the updating operations of − →

δ ⁽ⁱ⁾ and − → x ⁽ⁱ⁾ can be done in a distributed manner.

Now we describe Consort in detail. In the i ^th iteration, according to (6.2) we update Lagrange multipliers as follows:

∀u ∈ V, α ⁽ⁱ⁾ (u) = [α ⁽ⁱ⁻¹⁾ (u) + ηM ⁽ⁱ⁻¹⁾ _u ] ⁺ ,

∀(u, v) ∈ E k , β _k ⁽ⁱ⁾ (u, v) = [β _k ⁽ⁱ⁻¹⁾ (u, v) + ηC _(k,u,v) ⁽ⁱ⁻¹⁾ ] ⁺ ,

∀u ∈ V, μ ⁽ⁱ⁾ (u) = [μ ⁽ⁱ⁻¹⁾ (u) + ηG _u ⁽ⁱ⁻¹⁾ ] ⁺ ,

∀u ∈ V and v ∈ A(u), ω ⁽ⁱ⁾ (u, v)=[ω ⁽ⁱ⁻¹⁾ (u, v)+ηH ⁽ⁱ⁻¹⁾ _(u,v) ] ⁺ , where

M ⁽ⁱ⁻¹⁾ _u = 

k∈[1,K],u=s k

b ⁽ⁱ⁻¹⁾ _k (u) + 

k∈[1,K]



v∈R(u),u=s k

b ⁽ⁱ⁻¹⁾ _k (v) − C, C _(k,u,v) ⁽ⁱ⁻¹⁾ = r _k ⁽ⁱ⁻¹⁾ (u, v) − b ⁽ⁱ⁻¹⁾ _k (u) · p(u, v);

G ⁽ⁱ⁻¹⁾ _u = 

k∈[1,K]

b ⁽ⁱ⁻¹⁾ _k (u) − ϕ(u) H _(u,v) ⁽ⁱ⁻¹⁾ = 

k∈[1,K]

 b ⁽ⁱ⁻¹⁾ _k (u) − b ⁽ⁱ⁻¹⁾ _k (v) 

−θ(u, v),

After Lagrange multipliers α ⁽ⁱ⁾ (u), β _k ⁽ⁱ⁾ (u, v), μ ⁽ⁱ⁾ (u) and ω ⁽ⁱ⁾ (u, v) are computed, we need to update r ⁽ⁱ⁾ _k (u, v) and b ⁽ⁱ⁾ _k (v). According to (6.2), we need to com- pute − → x ⁽ⁱ⁾ = ((− → r ⁽ⁱ⁾ ) ^T , (− →

b ⁽ⁱ⁾ ) ^T ) ^T such that − → x ⁽ⁱ⁾ ∈ argmin _{→ − x ∈} − → X (L(− → x , − →

δ ⁽ⁱ⁾ ) +  _N

n=1  _n ||x n − x ⁽ⁱ⁻¹⁾ n || ² ). Based on (4.9), (4.10), (5.8) and (5.9), we know that the operation of minimizing L(− → x , − →

δ ⁽ⁱ⁾ ) over − → x ∈ − →

X can be decomposed to user behaviors and node behaviors. Therefore, the operation of minimizing (L(− → x , − →

δ ⁽ⁱ⁾ ) +  _N

n=1  _n ||x n − x ⁽ⁱ⁻¹⁾ n || ² ) over

−

→ x ∈ − →

X can also be decomposed to user behaviors and node behaviors, as  _N

n=1  _n ||x n − x ⁽ⁱ⁻¹⁾ n || ² is decomposable.

We first consider how user k updates its behavior − → r ⁽ⁱ⁾ _k in the i ^th iteration. Based on (4.9), (5.8) and (6.2) each user in both Systems 1 and 2 must solve

Minimize −U k (λ ⁽ⁱ⁾ _k ) + 

(u,v)∈E k

β ⁽ⁱ⁾ _k (u, v)r ⁽ⁱ⁾ _k (u, v), s.t. 

(u,v)∈E k

r ⁽ⁱ⁾ _k (u, v) − 

(w,u)∈E k

r _k ⁽ⁱ⁾ (w, u) = h ⁽ⁱ⁾ _k (u), u ∈ V k

over : r ⁽ⁱ⁾ _k (u, v) ∈ [0, C], ∀(u, v) ∈ E k .

Note that all the regulation terms  _n associated with r ⁽ⁱ⁾ _k (u, v) are set to 0, as the objective function is strictly convex already.

Based on the flow-path formulation technique in [33], we now consider an equivalent problem:

Minimize −U k ( 

π∈P

γ _k (π)) + 

π∈P

κ _k (π)γ _k (π),

over: 

π∈P

γ _k (π) ∈ [0, C],

where P is a set of single paths, γ _k (π) is the flow rate of user k following path π, and κ _k (π) = 

(u,v)∈π β _k ⁽ⁱ⁾ (u, v).

Obviously, solving this equivalent problem will result in that the min-cost path (with respect to β ⁽ⁱ⁾ _k (u, v)) is always chosen, i.e. a single path flow. We use Γ _k to denote the value of this single path flow. Thus this problem above is transformed to

Minimize − U k (Γ _k ) + κ ^min _k Γ _k , over: Γ _k ∈ [0, C], (6.4) where κ ^min _k is the cost of the min-cost path (with respect to β _k ⁽ⁱ⁾ (u, v)). Since U _k (·) is an increasing, strictly concave and continuously differentiable function, (6.4) can be easily solved.

r ⁽ⁱ⁾ _k (u, v) is set to the solution to (6.4) if (u, v) is on this min- cost path. Therefore, each user k can use a distributed shortest path algorithm (such as [7]) to compute its user behavior − → r ⁽ⁱ⁾ _k . We then consider how node u updates its behavior b ⁽ⁱ⁾ _k (u) in the i ^th iteration. For System 1, according to (4.10) and (6.2), node u should solve the following problem:

Minimize



α ⁽ⁱ⁾ (u) _u=s

k + 

v∈R(u),v=s k

α ⁽ⁱ⁾ (v) − 

(u,v)∈E k

β ⁽ⁱ⁾ _k (u, v)p(u, v) +μ ⁽ⁱ⁾ (u) + 

v∈A(u)

ω ⁽ⁱ⁾ (u, v) − 

v∈A(u)

ω ⁽ⁱ⁾ (v, u)  b ⁽ⁱ⁾ _k (u) +||b ⁽ⁱ⁾ _k (u) − b ⁽ⁱ⁻¹⁾ _k (u)|| ² ,

over: b ⁽ⁱ⁾ _k (u) ∈ [0, C], u ∈ V _k \ d _k .

Note that all the regulation terms associated with b ⁽ⁱ⁾ _k (u) are set to  > 0 to make the objective function strictly convex.

Otherwise, obviously oscillation will happen as the algorithm iterates due to its non-strict convexity. In other words, either 0 or C is always chosen as b ⁽ⁱ⁾ _k (u). As a result, the node broadcast rate oscillates as our algorithm iterates and is thus not smooth.

For System 2, according to (5.9) and (6.2), node u should solve the following problem:

Minimize



α ⁽ⁱ⁾ (u) _u=s

k + 

v∈R(u),v=s k

α ⁽ⁱ⁾ (v) − 

(u,v)∈E k

β _k ⁽ⁱ⁾ (u, v)p(u, v)

+ 

v∈A(u)

ω ⁽ⁱ⁾ (u, v) − 

v∈A(u)

ω ⁽ⁱ⁾ (v, u) + μ ⁽ⁱ⁾ (u) + σ _k (u)  b ⁽ⁱ⁾ _k (u) +||b ⁽ⁱ⁾ _k (u) − b ⁽ⁱ⁻¹⁾ _k (u)|| ² ,

over: b ⁽ⁱ⁾ _k (u) ∈ [0, C], u ∈ V k \ d k .

The average user behaviors and node behaviors till the i ^th iteration interval can be computed as follows:

ˆr ⁽ⁱ⁾ _k (u, v) = 1 i

 i−1 m=0

r ^(m) _k (u, v), (6.5)

and

ˆb ⁽ⁱ⁾ _k (u) = 1 i

 i−1 m=0

b ^(m) _k (u), i ≥ 1.

We now analyze what information is needed to execute

our distributed algorithm. We first analyze the computation of

Lagrange multipliers. For α ⁽ⁱ⁾ (u), node u needs its one-hop

neighbors’ and its own broadcast rates. For β _k ⁽ⁱ⁾ (u, v), node u needs r _k ⁽ⁱ⁾ (u, v) and b ⁽ⁱ⁾ _k (u). For μ ⁽ⁱ⁾ (u), node u needs its own b ⁽ⁱ⁾ _k (u). For ω ⁽ⁱ⁾ (u, v), u needs the broadcast rates of the nodes in its load balance area. Now we analyze the updates of user and node behaviors when the Lagrange multipliers have been obtained. Each user can use a distributed shortest path algorithm (such as [7]) to update user behavior. For node behaviors, each node should obtain the information from its one-hop neighbors and the nodes in its load balance area.

Through the analysis above, we know that no global iteration information is needed, and this analysis also implies that the iteration starting point {− → r ⁽⁰⁾ , − →

b ⁽⁰⁾ , − → α ⁽⁰⁾ , − →

β ⁽⁰⁾ , − → μ ⁽⁰⁾ , − → ω ⁽⁰⁾ } can be negotiated in a distributed manner. Note that the analysis above indicates that if a node’s load balance area is large (e.g. the load balance area is the set of H-hop neighbors and H is not small), it will introduce a lot of information exchanges. We will discuss this issue in Section 7.

C. Optimality and convergence analysis

Before using Theorem 6.1 to analyze the optimality and convergence, we need to prove the following two lemmas.

Lemma 6.1: Systems 1 and 2 satisfy the Slater condition. 2 Proof. We will construct the following user behavior and node behavior vector − →

x ^s for both Systems 1 and 2, which satisfies the Slater condition [5]. We use p ^min to denote min _(u,v)∈E p(u, v), ϕ ^min to denote min _u∈V ϕ(u), and θ ^min to denote min _u,v∈V θ(u, v). We let the information flow λ _k of each user k be equal to ^{(min{C,ϕ min} _{K|V |} ^{,θ min }−ι 1 )p min} −ι 2 , where ι ₁ is a small positive value such that min{C, ϕ ^min , θ ^min } − ι ₁ > 0 (note that ϕ ^min > 0 and θ ^min > 0), and ι ₂ is a very small positive value such that ^{(min{C,ϕ min} _{K|V |} ^{,θ min }−ι 1 )p min} − ι ₂ > 0 (note that p ^min > 0). Since all the nodes for each user k obtained in the node pre-selection process are reachable from the source of this user and can reach its destination, there must exist − → r _k such that r _k (u, v) ∈ (0, ^{(min{C,ϕ min} _{K|V |} ^{,θ min }−ι 1 )p min} − ι ₂ ), ∀(u, v) ∈ E k . Thus − → r is in the relative interior of its domain. If u ∈ V k \ d k , b _k (u) is set to ^{(min{C,ϕ min} _{K|V |} ^{,θ min }−ι 1 )} . Otherwise, b _k (u) is set to 0. This implies that b _k (u) ∈ (0, C), ∀u ∈ V k \ d k , k ∈ [1, K]. We can easily verify that based on the vector so constructed the broadcasting rate, coding, node max load and node load balance constraints are strictly feasible. Thus the Slater condition holds.

Lemma 6.2: The norms of subgradients of Systems 1 and 2 are bounded by a constant L(i.e.L = max → − x ∈ − →

X ||g(− → x )||).2 Proof. Since the domain of each element of − → x is [0, C], we know − →

X is a compact set. According to [25], we know that there exists a constant L such that L = max − → x ∈ − →

X ||g(− → x )||.

Theorem 6.2: The following properties hold for all i ≥ 1:

1) An upper bound on the amount of constraint violation of the vector ˆ x ⁽ⁱ⁾ for System 1 (System 2) is given by

||g(ˆx ⁽ⁱ⁾ ) ⁺ || ≤ B iη , where B = _ ² (f(− →

x ^s )−q( − →

δ ⁽⁰⁾ ))+max{|| − →

δ ⁽⁰⁾ ||, _ ¹ (f(− → x ^s )

−q( − →

δ ⁽⁰⁾ )+ ^ηL ₂ ² +KC ² |V |)+ηL} ( − →

x ^s is the vector con- structed in Lemma 6.1 and = min _j∈[1,ρ] (−g j (− →

x ^s ))).

2) An upper bound on f (ˆ x ⁽ⁱ⁾ ) is given by f (ˆx ⁽ⁱ⁾ ) ≤ f ^∗ + ||

−

→ δ ⁽⁰⁾ || ² 2iη + ηL ²

2 + KC ² |V |, where f ^∗ is the optimal value of the primal objective function of System 1(System 2). Note that we need to transform System 1 or 2 to the standard form in (6.1).

3) A lower bound on f (ˆ x ⁽ⁱ⁾ ) is given by f (ˆx ⁽ⁱ⁾ ) ≥ f ^∗ − B ²

iη . 2

Proof. According to our algorithm, we know that we have at most K|V | elements related to node behaviors and K elements related to user behaviors. Additionally, the value of  _n related to user behavior is set to 0, while the value of  _n related to node behavior is set to  > 0. Obviously, X n = C. According to Lemmas 6.1 and 6.2 and Theorem 6.1, we can easily prove Theorem 6.2 by substituting the related values above.

Theorem 6.2.1) states that the amount of feasibility violation of the primal solution so generated diminishes to zero at the rate ¹ _i as the number of iterations i increases. 2) and 3) imply that Consort converges to f ^∗ within error level ( ^ηL ₂ ² + KC ² |V |) with the same rate of ¹ _i . Note that we can adjust the values of η and  to obtain an arbitrarily small error level at the cost of slowing down the convergence rate.

7. D ISCUSSIONS

A. Practical issues

Integration with IEEE 802.11: Our solution tells each node u the optimized information rate r _k (u) = 

(u,v)∈E k r _k (u, v) and broadcast rate b _k (u) for user k. Although we assume a collision free MAC in this paper, we can use a similar scheme as in MORE to integrate our work into 802.11. Due to the space limit, we only describe the two differences compared with the method for integration with 802.11 used in MORE.

In an 802.11 wireless network, a node should be triggered to transmit only when it receives a packet, and should perform the transmission only when the 802.11 MAC permits. The first difference is that we use a different transmission credit computation formula: T X _k (u) = ^b _r ^{k (u)}

k (u) if r _k (u) > 0, or 0 otherwise. When u receives a packet for user k from a node upstream, it increments its associated credit counter by T X _k (u). The second difference is that when a transmission is allowed by the MAC, node u first uses a weighted round- robin algorithm [18] to select a user to serve. Each user k has a weight of ^{ r k (u)}

i∈[1,K] r i (u) . Then this node checks if the credit counter associated with this user is positive. If positive, the node creates a coded packet, broadcasts it, and then decrements its counter by one. Otherwise, the node re- selects a user to serve. If all the counters are non-positive, this node does not transmit.

Dynamic network and route updating: Our solution works

for static or slowly changing dynamic networks. Since the

network topology may change due to node mobility, fading,

and node failure, or user sessions may change, we need to

consider when to restart the iteration algorithm. We use the

following scheme. The timeline is divided into time slots of