3.3 Regression on the Grassmannian
3.3.1 Standard Geodesic Regression
First, the inner-product and the squared geodesic distance in Eq. (3.10) are adapted to G(p, n) by following Section 2.1.2. Next, given the auxiliary states, which are denoted as matricesX1 (initial point) andX2 (velocity), we can write the geodesic equation of Eq. (2.2) as a system of first-order dynamics:
˙
X1 =X2, and X˙2 =−X1(X>2X2) . (3.16)
For a point on G(p, n) it should further hold that (1)X1(r)>X1(r) =Ip and (2) the velocity at X1(r) needs to be orthogonal to that point, i.e., X1(r)>X2(r) = 0. If we enforce these
Algorithm 1 Standard Grassmannian geodesic regression (Std-GGR)
Input: {(ri,Yi)}iN=0−1,α and σ2
Output: X1(r0), X2(r0)
Set initial X1(r0) and X2(r0),e.g.,X1(r0) =Y0, and X2(r0) =0.
while not convergeddo
Solve Eqs. (3.16) withX1(r0) andX2(r0) forward forr ∈[r0, rN−1].
Solve ( ˙ λ1 =λ2X>2X2, λ1(rN−1+) = 0, ˙ λ2 =−λ1+X2(λ>2X1+X>1λ2), λ2(rN−1) = 0
backward with jump conditions
λ1(ri−) = λ1(ri+) − σ12∇X1(ri)d
2
g(X1(ri),Yi), and ∇X1(ri)d
2
g(X1(ri),Yi) computed as
−2 LogX1(ri)Yi. For multiple measurements at a given ri, the jump conditions for each mea-
surement are added up.
Compute gradients with respect to initial conditions:
∇X1(r0)E = −(In−X1(r0)X1(r0)>)λ1(r0−) +X2(r0)λ2(r0)>X1(r0),
∇X2(r0)E = 2αX2(r0)−(In−X1(r0)X1(r0)>)λ2(r0).
Use a line search with these gradients to updateX1(r0) andX2(r0) (see Algorithm 2).
end while
two constraints at the starting point r0, they will remain satisfied along the geodesic. This yields min Θ E(Θ) = α trX2(r0) > X2(r0) + 1 σ2 N−1 X i=0 d2g(X1(ri),Yi)
subject to X1(r0)>X1(r0) =Ip, X1(r0)>X2(r0) = 0 and Eq. (3.16) ,
(3.17)
with Θ = {Xi(r0)}2
i=1. Based on the adjoint method, we obtain the shooting solution to Eq. (3.17), listed in Alg. 1. Note that the jump conditions on λ1 involve the gradient of the residual term d2g(X1(ri),Yi) with respect to X1(ri), i.e., the base point of the residual on the fitted geodesic; this gradient is −2 LogX
1(ri)Yi (See next subsection). This problem of
fitting a geodesic is referred to as standard Grassmannian geodesic regression (Std-GGR). Residuals to curves on the Grassmannian. In Algorithm 1 we need the gradient of the residualsd2
g(X1(ri),Yi) with respect to the base point X1(ri) in order to compute the jump conditions for the adjoint variable λ1. The residual measures the squared geodesic distance
Algorithm 2 Grassmannian equivalent of xk+1 = xk−∆tg, where ∆t is the time step and g is the gradient.
Input: X1(r0), X2(r0), ∇X1(r0)E,∇X2(r0)E, ∆t
Output: Updated Xu1(r0) and Xu2(r0) Compute Xu2(r0) =X2(r0)−∆t∇X2(r0)E Compute Xu
1(r0) by flowing for ∆t along geodesic with initial condition (X1(r0),−∇X1(r0)E)
Transport Xu2(r0) along the geodesic connecting X1(r0) to Xu1(r0), using (3.23), resulting inXuT2 (r0)
Project updated initial velocity onto the tangent space (for consistency): Xu
2(r0)←(In− Xu
1(r0)Xu1(r0)>)X uT 2 (r0).
between the point X1(ri) on the fitted curve and the corresponding measurement Yi. To derive this gradient, we consider the constrained minimization problem of two points with exact matching: E(X1(r)) = Z r1 r0 tr X2(r)>X2(r) dr, s.t. X1(r0) =Y0, X1(r1) =Y1, and X1(r0)>X1(r0) =Ip , (3.18)
with X2(r) = (In−X1(r)X1(r)>)C. We know that the squared distance can be formulated asd2
g(Y0,Y1) = minX1(r)E(X1(r)) forr0 = 0 andr1 = 1. After adding the constraint on the form ofX2(r) via the time-dependent adjoint variableλand the constraintX1(0)>X1(0) =Ip via λc, we obtain (by taking variations), among other terms, the optimality condition
(2C>−λ>)(In−X1(r)X1(r)>) = 0 , (3.19) and another optimality condition for a free initial condition4 X1(0)
∇X1(0)E =−λ(0) +X1(0)(λ >
c +λc) =0 . (3.20)
4Note that technically we started withX1(r0) =Y0, i.e., this condition would not be free and we would not need to consider variations ofX1(r0). However, the goal is to compute the energy gradient with respect to the initial conditionX1(r0). Consequentially, the variation of the energy with respect to it allows computing
Left-multiplication by X1(0)> yieldsλc+λ>c =X>1(0)λ(0) which we can use to obtain, upon back-substitution into Eq. (3.20),
−(In−X1(0)X1(0)>)λ(0) =0 . (3.21) Using Eq. (3.19) and the above expression for X2(r), we can obtain
∇X1(0)E =−(In−X1(0)X1(0) >
)λ(0) =−2X2(0) , (3.22) with X2(0) being the tangent vector at X1(0) such that ExpX1(0)(X2(0)) = X1(1). This tangent vector can be computed via the Log-map presented in Section 2.1.2, i.e., X2(0) = LogX1(0)(X1(1)).
Line search on the Grassmannian. Performing a line search on the Grassmann manifold in Algorithm 1 is not as straightforward as in Euclidean space since we need to assure that the constraints forX1(r0) and X2(r0) are fulfilled for any given step. In particular, changing X1(r0) will change the associated tangent vector X2(r0). Once, we have updated X1(r0) to Xu1(r0) by moving along the geodesic defined by X1(r0) and the gradient of the energy with respect to this initial point, i.e., ∇X1(r0)E, we can transport the tangent X2(r0) to
Xu1(r0) using the closed form solution for parallel transport of [Edelman et al., 1998] (cf. Section 2.1.2). In particular, Xu2(r0) = [X1(r0)V U] −sintΣ costΣ U>+ (In−UU>)X2(r0) (3.23)
where H=UΣV> is the compact SVD of the tangent vector atX1(r0) along the geodesic connecting X1(r0) and Xu1(r0). Note that t = 1 in Eq. (3.23). Algorithm 2 lists the line search procedure in full technical detail.
Note: when implementing Algorithm 1 (the same holds for Algorithm 3) it is important to pay attention to the ordering of the matrix multiplications, as performing them in an appropriate order will reduce time and memory complexity.
3.3.2 Time-Warped Regression
Since the concept of time-warped geodesic regression is generic for Riemannian manifolds, specialization to the Grassmannian is straightforward. The Std-GGR solution is used during the alternating optimization steps. By choosing the generalized logistic function of Eq. (3.13) to account for saturations of scalar-valued outputs, the time-warped model on G(p, n) can be used to capture saturation effects for which standard geodesic regression is not sensible. This strategy is referred to as time-warped Grassmannian geodesic regression (TW-GGR).
3.3.3 Cubic Spline Regression
To enable cubic spline regression on the Grassmannian, we follow Section 3.2.3 and add the external force X3. In other words, we represent an acceleration-controlled curve X1(r) onG(p, n) using a dynamic system with states (X1,X2,X3) such that
X2 = ˙X1, and X3 = ˙X2+X1(X>2X2) . (3.24) If X3 = 0, the second equation is reduced to the geodesic equation of Eq. (2.2), i.e., the geodesic is acceleration-free. To obtain an acceleration-controlled curve, we need to solve
min Θ E(Θ) = 1 2 Z 1 0 trX>3X3 dr s.t. X>1X1 =Ip, X>1X2 =0, and Eq. (3.24) (3.25)
only need to enforceX>1X1 =Ip initially. This can be seen by taking the derivative ofX>1X1 with respect to r, i.e.,
d drX
>
1X1 = ˙X>1X1+X>1X˙1 =X>2X1+X>1X2 =0 . (3.26)
In other words, if X1(0)>X1(0) = Ip holds, then X>1X1 = Ip holds for all r by enforcing X>1X2 =0 at all time points. Hence, Equation (3.25) can be rewritten as
min Θ E(Θ) = 1 2 Z 1 0 tr X>3X3 dr s.t.X>1X2 =0,Eq. (3.24), andX1(0)>X1(0) =Ip . (3.27)
For the first three time-evolving constraints, we introduce three time-dependent adjoint variables, λ1, λ2, and λ3, to convert the constrained problem to an unconstrained one. We then take the variations with respect to the state variables which results in the following system of adjoint equations:
−λ˙>1 + (X>2X2)λ>2 +λ3X>2 =0, −λ˙>2 −λ>1 +X>1λ2X>2 +λ>2X1X>2 +λ>3X>1 =0, X>3 −λ>2 =0 . (3.28)
UsingX3 =λ2 and settingX4 =λ1, we can rewrite the adjoint equations (3.28) as
˙ X>4 = (X>2X2)X>3 +λ3X>2, (3.29) ˙ X>3 =−X>4 +X>1X3X>2 +X > 3X1X>2 +λ > 3X > 1 . (3.30)
Next, we derive the form ofX3 using the dynamic constraints. By taking the time-derivative of X>1X2 we get d drX > 1X2 = ˙X>1X2+X>1X˙ 2 = ˙X1>X2+X>1(X3−X1X>2X2) =X>2X2+X>1X3−X>2X2 =X>1X3 . (3.31)
However, the constraintX>1X2 =0implies drdX1>X2 =0which yields X>1X3 =0. It further implies drdX>1X3 =0 and we get, as a side result,
X>2X3+X>1X˙3 =0 . (3.32)
We can then use X>1X3 =0 to simplify Eq. (3.30) to
−X˙>3 −X4>+λ>3X>1 =0 ⇔ −X˙3−X4+X1λ3 =0 . (3.33)
Upon left-multiplication of Eq. (3.33) by X1 we obtain the expression for λ3 as
λ3 =X>1X˙3+X>1X4 (3.32)
= X>1X4−X>2X3 . (3.34)
Substituting λ3 into Eq. (3.33) yields the evolution equation for ˙X3 as
˙
X3 =−X4+X1X>1X4−X1X>2X3 (3.35)
and substituting λ3 in Eq. (3.29) yields the evolution equation for ˙X4
˙
In summary, the system of equations for shooting cubic curves on G(p, n) are ˙ X1 =X2, X˙2 =X3−X1X>2X2, ˙ X3 =−X4+X1X>1X4−X1X>2X3, ˙ X4 =X3X>2X2+X2X>4X1−X2X>3X2 . (3.37)
It is important to note that X1 does not follow a geodesic path under non-zero force X3. Hence, the constraints X1(r)>X1(r) = Ip and X1(r)>X2(r) = 0 should be enforced at every instance of r to keep the path on the manifold. However, we showed that enforcing X1(r)>X2(r) =0at all times already guarantees thatX1(r)>X1(r) =Ipif this holds initially at r = 0. Also, X1(r)>X2(r) =0 implies that X1(r)>X3(r) = 0. By using this fact during relaxation, the constraints are already implicitly captured in Eqs. (3.37). Subsequently, for shooting we only need to guarantee that all these constraints hold initially. To get a cubic spline curve, we follow Section 3.1.3 and introduce control points {rc}Cc=1, which divide the support of the independent variable into several intervals Ic. The first three states should be continuous at the control points, but the state X4 is allowed to jump. Hence, the spline regression problem on G(p, n) becomes,cf. Eq. (3.7),
min Θ E(Θ) =α Z rN−1 r0 tr X>3X3 dr + 1 σ2 N−1 X i=0 d2g(X1(ri),Yi) subject to X1(r0)>X1(r0) = Ip, X1(r0)>X2(r0) =0, X1(r0)>X3(r0) =0,
X1,X2,X3 are continuous at {rc}Cc=1, and Eqs. (3.37) hold in each Ic,
(3.38)
with Θ = {{Xi(r0)}4i=1,{X4(rc+)}Cc=1}. Alg. 3 lists the shooting solution to Eq. (3.38), referred to as cubic-spline Grassmannian geodesic regression (CS-GGR).
Algorithm 3 Cubic-spline Grassmannian geodesic regression (CS-GGR)
Input: {(ri,Yi)}iN=0−1,{rc}Cc=1,α and σ2
Output: X1(r0), X2(r0),X3(r0),X4(r0),{X4(r+c)}Cc=1
Set initial X1(r0) asY0 for example, andX2(r0),X3(r0),X4(r0),{X4(r+c)}Cc=1 as zero matrices.
while not convergeddo
Solve Eq. (3.37) forward in each interval with X1(r0), X2(r0), X3(r0), X4(r0), {X4(rc+)}Cc=1,
and {X1(r+c) =X1(rc−), X2(r+c) =X2(rc−), X3(r+c) =X3(r−c)}Cc=1. Solve ˙ λ1 =λ2X>2X2−λ3(X>4X1−X3>X2)−X4(λ3>X1+X>2λ4), ˙ λ2 =−λ1+X2(λ>2X1+X>1λ2−λ4>X3−X>3λ4) +X3(λ>3X1+X>2λ4) +λ4(−X>1X4+X>2X3), ˙ λ3 =−λ2−λ4X>2X2+X2(X>1λ3+λ>4X2) + 2αX3, ˙ λ4 =λ3−X1(X1>λ3+λ>4X2) backward withλ1(rN−1) =λ2(rN−1) =λ3(rN−1) =λ4(rN−1) =λ4(rc−) = 0, and
{λ1(rc−) =λ1(rc+),λ2(r−c) =λ2(rc+),λ3(r−c) =λ3(rc+)}Cc=1, as well as jump conditionsλ1(ri−) = λ1(ri+)−σ12∇X1(ri)d
2
g(X1(ri),Yi), and∇X1(ri)d
2
g(X1(ri),Yi) computed as−2 LogX1(ri)Yi. For multiple measurements at a givenri, the jump conditions for each measurement are added up. Compute gradients with respect to initial conditions and the fourth state at control points:
∇X1(r0)E =−(In−X1(r0)X1(r0)>)λ1(r0−) +X2(r0)λ2(r0)>X1(r0) +X3(r0)λ3(r0)>X1(r0), ∇X2(r0)E =−(In−X1(r0)X1(r0)>)λ2(r0), ∇X 3(r0)E =−(In−X1(r0)X1(r0) >)λ 3(r0), ∇X4(r0)E =−λ4(r0), ∇X4(r+ c)E =−λ4(r + c), c= 1...C .
Use a line search with these gradients to update X1(r0), X2(r0), X3(r0), X4(r0), and
{X4(rc+)}Cc=1.
end while