Distributed low-rank matrix completion

The ability to recover a low-rank matrix from a subset of its entries is the leitmotif of recent advances for localization of wireless sensors [108], Internet traffic analysis [79], [160], and preference modeling for recommender systems [6]. In the low-rank matrix completion prob- lem, given a limited number of (possibly) noise corrupted entries of a low-rank matrix X, the goal is to recover the entire matrix while denoising the observed entries, and accurately imputing the missing ones.

4.4 Applications 116

Algorithm 7 : DRPCA algorithm per agent n ∈ N

input Yn, λ∗, λ1, c, µ

initialize An[1] = 0Fn×T, O[0] = 0T ×ρ, and Ln[1], Qn[1] at random.

for k = 1, 2,. . . do

Receive {Qm[k]} from neighbors m ∈ Jn

[S1] Update local dual variables:

On[k] = On[k − 1] + µPm∈Jn(Qn[k] − Qm[k])

[S2] Update first group of local primal variables:

Gn[k + 1] := (Yn−An[k])0Ln[k]−On[k]+cPm∈Jn(Qn[k]+Qm[k])

Qn[k + 1] = Gn[k + 1] [L0n[k]Ln[k] + (λ∗/N + 2c|Jn|)Iρ] −1

[S2] Update second group of local primal variables:

Ln[k + 1] = (Yn− An[k]) Qn[k + 1] × [Q0n[k + 1]Qn[k + 1] + λ∗Iρ]−1

[S3] Update third group of local primal variables: An[k + 1] = Sλ1(Yn− Ln[k + 1]Q 0 n[k + 1]) Broadcast {Qn[k + 1]} to neighbors m ∈ Jn end for return An, Qn, Ln

In the network setting envisioned here, agent n ∈ N has available Ln incomplete and

noise-corrupted rows of Y := [Y0₁, . . . , Y_N0 ]0. Local data can thus be modeled as PΩn(Yn) = P_Ωn(Xn+ Vn). Relying on in-network processing, agents aim at completing their own rows

by forming the global estimator

ˆ X = arg min X N X n=1  1 2kPΩn(Yn− Xn)k 2 F + λ∗ NkXk∗  (4.19)

which exploits the low-rank property of X through nuclear-norm regularization. Estimator (4.19) was proposed in [26], and solved centrally whereby all data PΩn(Yn) is available to feed e.g., an off-the-shelf semidefinite programming (SDP) solver. The general estimator in (P1) reduces to (4.19) upon setting R = 0L×F and λ1 = 0. Hence, it is possible to derive a

decentralized algorithm for low-rank matrix completion by specializing Algorithm 5 to the setting here.

Before discussing the algorithmic details, a brief parenthesis is in order to touch upon properties of local sampling operators. Operator PΩn is a linear orthogonal projector, since

4.4 Applications 117

it projects its matrix argument onto the subspace Ψn := {Z ∈ RLn×T : supp(Z) ∈ Ωn}

of matrices with support contained in Ωn. Linearity of PΩn implies that vec(PΩn(Z)) = AΩnvec(Z), where AΩn ∈ R

Ln×T _{is a symmetric and idempotent projection matrix that will} prove handy later on. To characterize AΩn, introduce an Ln× T masking matrix Ωnwhose (l, t)-th entry equals one when (l, t) ∈ Ωn, and zero otherwise. Since PΩn(Z) = Ωn Z, from standard properties of the vec(·) operator it follows that AΩn = diag(vec(Ωn)). Distributed Matrix Completion (DMC) Algorithm. Going back to the general de- centralized formulation in (P4), since there is no sparse component A in the matrix comple- tion problem (4.19), the only constraints that remain are Qn= Qm, m ∈ Jn, n ∈ N . These

correspond to the dual variables On[k] per agent, and are updated as shown in Algorithm 8.

In the absence of {An}n∈N and the auxiliary variables {Bn}n∈N, it suffices to cycle

over two groups of primal variables to arrive at the DMC iterations. The primal variable updates can be readily obtained by capitalizing on the properties of the vec(·) operator. In particular, Algorithm 5 indicates that the recursions for Qn are given by [let q := vec(Q0)]

qn[k + 1] = arg min q  1 2kAΩn(vec(Yn) − (I ⊗ Ln[k])qk 2_{) +} λ∗ 2Nkqk 2_{+ hvec(O} n0[k]), qi + c X m∈Jn q − vec(Qn 0 [k] + Qm0[k]) 2 2 . (4.20)

Likewise, Lnis updated by solving the following subproblem per iteration (let l := vec(L))

ln[k + 1]=arg min l  1 2kAΩn(vec(Yn)−(Qn[k + 1] ⊗ ILn)l) k 2₊λ∗ 2 klk 2  . (4.21)

Both (4.20) and (4.21) are unconstrained convex quadratic problems, which admit the closed-form solutions tabulated under Algorithm 8.

The per-agent computational complexity of the DMC algorithm is dominated by re- peated inversions of ρ × ρ and ρLn× ρLn matrices to obtain En[k + 1] and Dn[k + 1],

respectively, and matrix multiplications to update Qn[k + 1] and Ln[k + 1] (cf. Algorithm

8). Notice that En[k + 1] ∈ RρT ×ρT has block-diagonal structure with blocks of size ρ × ρ.

Overall, the per-iteration complexity across the network is upper bounded by O(ρ3_{N T ),}

which grows linearly with the network size. This is affordable since in practice ρ is typically small for a number of applications of interest (cf. the low-rank assumption). In addition,

4.4 Applications 118

Algorithm 8 : DMC algorithm per agent n ∈ N

Input Yn, Ωn, AΩn, λ∗, c, µ

Initialize O[0] = 0T ×ρ, and Ln[1], Qn[1] at random

for k = 1, 2,. . . do

Receive {Qm[k]} from neighbors m ∈ Jn

[S1] Update local dual variables:

On[k] = On[k − 1] + µPm∈Jn(Qn[k] − Qm[k])

[S2] Update first group of local primal variables:

En[k + 1] = {(IT⊗ L0n[k])AΩn(IT ⊗ Ln[k]) + (λ∗/N + 2c|Jn|)IρT} −1

Gn[k + 1] := (IT⊗ L0n[k])AΩnvec(Yn) − vec(O 0 n[k]) + cvec( P m∈Jn(Q 0 n[k] + Q0m[k])) Q0_n[k + 1] = unvec En[k + 1]Gn[k + 1]

[S3] Update second group of local primal variables: Dn[k + 1]:= {(Q0n[k + 1] ⊗ ILn)AΩn(Qn[k + 1] ⊗ ILn)+λ∗IρLn} −1 Ln[k + 1] = unvec (Dn[k + 1] (Q0n[k + 1] ⊗ ILn)AΩnvec(Yn)) Broadcast {Qn[k + 1]} to neighbors m ∈ Jn end for Return Qn, Ln

Ln, the number of row vectors acquired per agent, and T , the number of time instants for

data collection, can be controlled by the designer to accommodate a prescribed maximum computational complexity. One can also benefit from the decomposability of (4.21) and (4.20) across rows of L and Q, respectively, and parallelize the row updates. This way, one only needs to invert ρ × ρ matrices. On a per-iteration basis, network agents communicate their updated local estimates Qn[k] only with their neighbors, in order to carry out the

updates of primal and dual variables during the next iteration. In terms of communication cost, Qn[k] is a T × ρ matrix and its transmission does not incur significant overhead for

small values of ρ. Observe that the dual variables On[k] need not be exchanged, and the

4.5 Numerical Tests 119