Recall that the event Enamounts to non-existence of MLE for finite-dimensional logistic
regression models (see Albert and Anderson (1984)).
Now let us prove the validity of condition (3) in Candès and Sur (2018), which is required for the validity of Theorem 1 in that paper. In our case, such condition amounts to lim n→∞var α 0 TnX(Tn) = limn→∞α 0 TnΣTnαTn = kβ ∗k2 K,
but this directly follows from Theorem 6E of Parzen (1959). Since limn→∞pn/n = κ >
h(β∗0, kβ∗k2
K) we apply Theorem 1 in Candès and Sur (2018) to deduce limn→∞P(En) =
1.
Now we define the auxiliary sequence of events e
En= {There exists α ∈ Rpn : α0xi(Tn) > 0, if yi = 1; α0xi(Tn) < 0, if yi = 0},
with strict inequalities. Assume that eEnhappens so that there exists a separating hyper-
plane defined by α ∈ Rpn. Then, in the same spirit as in the proof of Theorem 4.4, it is
possible to show that if ˆβm,n = mPpj=1n αjK(·, tj) ∈ HK, then limm→∞L( ˆβm,n, 0) = 0,
where L(β, β0) is the log-likelihood function. As a consequence, for all n, if eEnhappens,
then the MLE for the RKHS functional logistic regression model does not exist. The result follows from the fact that P(En) = P( eEn) and the events α0xi(Tn) = 0 have
probability zero. Because we are assuming that the process does not have degenerate marginals.
Remark. Theorem 1 in Candès and Sur (2018) is a remarkable result. It applies to logistic finite-dimensional regression models with a number p of covariables, which is assumed to grow to infinity with the sample size n, in such a way that p/n → κ. Of course, the sample is given by data (xi, yi), i = 1, . . . , n. Essentially the result
establishes that there is a critical value such that, if κ is smaller than such critical value, one has limn,p→∞P(MLE exists) = 1; otherwise we have limn,p→∞P(MLE exists) = 0. Such critical value is given in terms of a function h (which is mentioned in the proof of the previous result) whose definition is as follows. Let us use the notation ( eY , V ) ∼ Fβ0,γ0whenever ( eY , V )
d
= (Y, Y X), for eY = 2Y −1 (note that, in the notation of Candès and Sur (2018), the model is defined for the case that the variable Y is coded in {−1, 1}), β0, γ0∈ R, γ0 ≥ 0 and where X ∼ N (0, 1) and P( eY = 1|X) = (1 + exp{−β0− γ0X})−1.
Now, define h(β0, γ0) = mint0,t1∈RE[(t0Y +te 1V −Z)
2
+], where Z ∼ N (0, 1) independent
of ( eY , V ) and x+ = max{x, 0}. Then, Theorem 1 in Candès and Sur (2018) proves that
the above mentioned critical value for κ is precisely h(β0, γ0).
4.3 The estimation ofβ
in practice
The problem of non-existence of the MLE can be circumvented if the goal is variable selection. The main idea behind the proof of Theorem 4.6 is that one can approximate
the functional model with finite approximations as those in (4.6) with p increasing as fast as desired. Therefore, if we constrain p to be less than a finite fixed value, Theorem 4.6 does not apply.
In order to sort out the non-existence problem for a given sample (due to the SC prop- erty), it would be enough to use a finite-dimensional estimator that is always defined, even for linearly separable samples. As mentioned, an extensive study of existence and uniqueness conditions of the MLE for multiple logistic regression can be found in the paper of Albert and Anderson (1984). We suggest to use Firth’s estimator, firstly pro- posed by Firth (1993), which is always finite and unique.
The author initially proposed this approach to reduce the bias of the standard MLE, but afterwards it was used to obtain ML estimators for linearly separable samples (see e.g. Heinze and Schemper (2002), where the technique is presented in a quite accessible manner). Besides, the reduction of the bias leads to better results in practice. The gen- eral idea of Firth’s procedure is not to use the original sample responses (y1, . . . , yn) to
compute the usual score equations one must solve to compute the MLE, but a mod- ification of them. Each response yi is split into two new responses (1 + hi/2)yi and
(hi/2)(1 − yi), where the coefficients hi go to zero with the sample size. The idea be-
hind this duplication is to avoid the problem of the linear separability of the sample (previously discussed), which leads to the non-existence of the MLE. It is easy to see that in the new modified sample with duplicated observations there is always overlap- ping between the two classes, so ML can be safely applied. The specific coefficients hi are the diagonal elements of the matrix W1/2xT(x0TW xT)−1W1/2, W being the di-
agonal matrix with elements pβ,β0(xi)(1 − pβ,β0(xi)) and xT the matrix whose rows
are (1, xi(t1), . . . , xi(tp)). This estimator is implemented in the “brglm” function of the
brglm R-package (see Kosmidis (2017)).
With this procedure we obtain estimators bβ0, . . . , bβp. An interesting observation is that
the value of the independent term bβ0 can be used to estimate the Bayes error of any
homoscedastic Gaussian problem with equiprobable classes (p = 1/2), as we discussed at the end of Section 4.1.1.
4.3.1 Greedy “max-max” algorithm
In view of the previous discussion, the objective is to find the points T = (t1, . . . , tp) ∈
[0, 1]p and the coefficients (β0, β1, . . . , βp) ∈ Rp+1, for a fixed p, that maximize the
log-likelihood associated with model of Equation (4.6).
We propose an iterative algorithm in which the points and the coefficients are updated alternatively. This approach is reminiscent of the well-known Expectation-Maximization (EM) technique, which is typically applied to estimate mixtures in supervised classi- fication (see e.g. Bishop (2016, Chapter 9)). In general, the EM algorithm is used to
4.3 The estimation of β in practice
compute ML estimators for models with non-observable data. In our setting, these non- observable parameters would correspond to the points t1, . . . , tp. Estimating the βj’s
given a set of tj’s is straightforward (via Firth’s approach), and once the parameters
βj are known, the points tj can be obtained maximizing the log-likelihood over [0, 1]p.
The algorithm is as follows: first the coefficients βj are randomly initialized, and then we iterate the following steps until convergence.
• (Maximization 1) Compute the set of points T ∈ [0, 1]p that maximizes the
log-likelihood function for the current set of coefficients β0, . . . , βp.
• (Maximization 2) Then, use the just obtained points T to compute the MLE of the model (via Firth’s estimator).
It is also suitable to start initializing the points tj, and then start the iterations in the
second step. Besides, in practice the maximization over the continuous set T ∈ [0, 1]p is unfeasible, so it should be made using some grid. This is not usually an issue, since the sample curves are typically provided in a discretized fashion. Then one could directly search the points in the grid provided by the sample.
However, the proposed algorithm only ensures the convergence to a local maximum, and this maximum strongly depends on the initial (random) points of the algorithm. Besides, the number of local maxima of the likelihood function likely increases with the dimension of the search space. That is, the accuracy of the results might deteriorate when p increases. A possible solution is to replicate the execution several times for the same sample and keep the parameters that give a maximum value of the likelihood. However, this could be computationally expensive depending on the dimension. Then, we suggest to adapt the greedy EM methodology proposed in Verbeek et al. (2003) for Gaussian mixtures. The idea is to start with a model of dimension one (p = 1 in Equation (4.6)) to estimate bt1. Then, once this first point is fixed, add a second random pointet2 and start again the maximization iterations, now with p = 2, using
as starting points (bt1,et2). The algorithm continues adding points until it reaches the
desired dimension p. This approach should work better than simply initializing all the points tj at random, since only one random point is added in each step. In addition, for small values of p it is more likely to obtain meaningful points, since the likelihood function should have less local maxima.
In practical problems, it is also important to determine how many points p one should retain. The common approach is to fix this valuep by cross-validation, whenever it isb possible. Another reasonable approach is to stop when the difference between the likeli- hoods obtained with p − 1 and p points is less than a threshold . This was the method used in Chapter 3, where some techniques to determine the value were addressed. Once that the parameters are estimated for a given sample, one can also approximate
the β function in H(K) as b β(·) = b p X j=1 b βjK(btj, ·).
When the function K is not known, a feasible alternative is to replace K(btj, ·) with the
empirical covariance function bK(btj, ·). However, in this case the estimation bβ would not
be in H(K) with probability 1 (since the sample curves Xi used to compute bK do not belong to H(K)). A possible solution is to regularize the trajectories before computing
b
K, to force them to belong to H(K) (in the same spirit as in Chapter 2).
4.3.2 Sequential maximum likelihood
The greedy approach we have described above directly suggests another greedy algo- rithm to compute both the points tj and the coefficients βj. The idea is to exchange
the execution of the iterative algorithm by the direct maximization of the likelihood function. As in the previous case, we will need also a grid over [0, 1] to search the points tj. The procedure would be as follows:
1. For each t on the grid, we fit the logistic model of Equation (4.6) with p = 1. The log-likelihood achieved for this t at the ML estimators bβ0 and bβ1 is stored
in `1(t). Then, the first point bt1 is fixed as the point at which `1(t) achieves its
maximum value.
2. Oncebt1 has been selected, for each t in the grid we fit the model
P(Y = 1|X) = 1 + exp n β0+ β1X(bt1) + β2X(t2) o−1 .
As in the previous step, `2(t) would be be the likelihood achieved at bβ0, bβ1 and
b
β2 andbt2 the maximum of `2(t).
3. We proceed in the same way until a suitable number of points p has been selected. The number of points to select and the complete estimation of the β function can be obtained as in the previous proposal.
With this greedy approach the dependence on the random initial points is erased. However, it may be more computationally expensive, specially if the size of the grid is large.