In this section we cite some known, and present some new results on constant weight codes with which we attempt to explain the observed performance of constant weight almost disjunct matrices. The following well-known result of [129] has been the basis of a large number of construction of testing matrices.
129
Proposition 3 An(m, n, d, w) constant weight binary codeCprovides ak-disjunct ma- trix, where k=ww−−d/12.
This proposition implies that a group testing scheme can be obtained from constant weight codes with large distance. However, the set of codewords of weight 5 in a [63,51] BCH code forms a (63,1890,3,5) constant weight code with distance d = 5, so Proposition 3 clearly fails to explain the performance of the group testing scheme obtained from this code.
Extending the theory of Kautz-Singleton to almost-disjunct matrices, [133] provides a bound on the false-positive probability of a constant weight code matrix in terms of its distance distribution. Define theaverage distance Dof a codeC:
D(C) = 1
|C|minx∈C
X y∈C
dH(x, y).
HeredHdenotes the Hamming distance. Define also the second-moment of the distance distribution as follows: D2(C) = 1 |C|2 X x,y∈C dH(x,y)2.
One of the main results of [133] is the following theorem.
Theorem 10 Let C be a constant weight binary code C of size n, minimum distance d and average distance D such that every codeword has length m and weightw. The test matrix obtained from the code is k-disjunct() for the largest k such that the inequality
d≥D− 3(w−k(w−D/2))
2 (ln 1/)(2k(w−D/2) +w) holds true.
Paper [133] also provides a more refined estimate that relies on the second moment of the distance distribution.
Theorem 11 Let Cbe a constant-weight(m, N, d, w)binary code with average distance D and the second moment of the distance distribution D2. The test matrix obtained
Table 8.1: Minimum distance and average distance
Parameters d=dmin D=davg D2
m= 31, w= 6 6 9.6894 96.7742 m= 63, w= 3 4 5.7231 33.1797 m= 63, w= 5 6 9.2112 86.1239 m= 63, w= 7 8 12.4481 157.3620 m= 63, w= 9 10 15.4357 241.8802 m= 127, w= 3 4 5.8605 34.5917 m= 127, w= 5 6 9.6050 93.0134
Table 8.2: Estimates of the error probabilityfrom Theorem 10
Parameters k= 1 k= 2 k= 3 k= 4 k= 5 k= 6 k= 7 k= 8 k= 9 k= 10 m= 31, w= 6 0.101 0.354 0.671 0.906 0.998 0.963 0.848 0.699 0.550 0.416 m= 63, w= 3 0.013 0.026 0.048 0.080 0.122 0.175 0.238 0.308 0.384 0.463 m= 63, w= 5 0.032 0.080 0.157 0.261 0.384 0.514 0.642 0.756 0.851 0.923 m= 63, w= 7 0.047 0.138 0.284 0.463 0.644 0.800 0.914 0.980 1.000 0.981 m= 63, w= 9 0.058 0.197 0.414 0.649 0.842 0.961 1.000 0.972 0.896 0.794 m= 127, w= 3 0.012 0.017 0.025 0.035 0.046 0.061 0.078 0.097 0.119 0.143 m= 127, w= 5 0.028 0.047 0.073 0.106 0.146 0.193 0.246 0.303 0.364 0.427
Table 8.3: Estimates of the error probabilityfrom Theorem 11
Parameters k= 1 k= 2 k= 3 k= 4 k= 5 k= 6 k= 7 k= 8 k= 9 k= 10 m= 31, w= 6 0.043 0.164 0.431 0.781 0.993 0.896 0.578 0.271 0.093 0.023 m= 63, w= 3 0.012 0.026 0.048 0.079 0.120 0.172 0.233 0.302 0.377 0.454 m= 63, w= 5 0.022 0.048 0.091 0.158 0.249 0.362 0.493 0.628 0.758 0.867 m= 63, w= 7 0.024 0.060 0.130 0.246 0.410 0.607 0.802 0.945 1.000 0.951 m= 63, w= 9 0.024 0.072 0.181 0.376 0.641 0.891 1.000 0.895 0.632 0.347 m= 127, w= 3 0.012 0.018 0.025 0.035 0.046 0.060 0.077 0.096 0.118 0.142 m= 127, w= 5 0.023 0.033 0.048 0.066 0.089 0.116 0.150 0.189 0.234 0.285
from the code is k-disjunct() for the largest k such that the inequality
d≥D+ 3t(D2−D 2) 2(w−t(w−D/2)) − 3(w−t(w−D/2)) ln 1/ (8.1) holds true.
In order to compute the estimates based on these theorems, we computed the dis- tance distributions of a number of constant weight codes obtained from low-weight codewords of the BCH codes of length 63 and 127 mentioned above, and found D and
D2 for these codes. The results are listed in Table 8.1. Using these values, we can find the estimates of the error probability given by Theorems 10 and 11. The results
131 are summarized in Tables 8.2 and 8.3, respectively. Although they represent a large improvement over the initial estimates of Kautz-Singleton, they still do not match the performance in actual experiments. For example, with codewords of weight 5 in a BCH code of length 127, even with 3 defectives the predicted false positive probability is 0.0479, whereas the number of false positives in the experiments is close to zero.
Next we will show that better estimates can be achieved in many cases. To formulate the result we need to define the dual distance of a constant weight code. The distance distribution of a constant weight code C(m, n, d, w) is a set of numbers b0, b1, . . . , bw,
where
bi=
1
|C||{(x, y)∈ C
2 :w− |supp(x)∩supp(y)|=i}| (8.2)
fori= 0,1, . . . , w. Note that b0 = 1.The dual distanced0 of C is defined as
d0(C) = minnj≥1 :b0j := 1 |C| w X i=0 biQj(i)>0 o ,
where Qj(i) is the value of the Hahn polynomial of degree j; see [128].
Now we are ready to state the main theorem in our analysis.
Theorem 12 Let Cbe an(m, n, d, w)constant weight code with dual distanced0 and let w < m/2. Letk be the maximum number of defective items and suppose thatk < m/w. For any even ` < d0 the probability of a false positive test result for the group testing scheme constructed from C is bounded above as
≤B(`, k) e`(m−w) 2(m−kw)2 `/2X`/2 i=0 (m−w)` 2ew2 i , (8.3)
where B(`, k) = min{(18`k)`/2, k`}. In addition, if m ≥ max{4w2k/`2, w+ 2ew2/`}, then ≤k 2` 2(m−w) (log`)w(m−kw) ` . (8.4)
In the case of `= 2 we have
< k m−1
(m−w)2
Table 8.4: Estimates of the error probability from (8.5), Thm. 12 Parameters k= 1 k= 2 k= 3 k= 4 k= 5 k= 6 k= 7 k= 8 k= 9 k= 10 m= 63, w= 3 0.016 0.036 0.060 0.089 0.126 0.172 0.230 0.305 0.403 0.533 m= 63, w= 5 0.016 0.039 0.070 0.117 0.188 0.299 0.484 – – – m= 63, w= 7 0.016 0.042 0.086 0.165 0.322 0.688 – – – – m= 127, w= 3 0.008 0.017 0.026 0.037 0.049 0.062 0.076 0.092 0.110 0.130 m= 127, w= 5 0.008 0.017 0.028 0.041 0.057 0.075 0.098 0.125 0.158 0.199 m= 127, w= 7 0.008 0.018 0.030 0.046 0.067 0.095 0.131 0.181 0.251 0.352
Proof: (outline) The ideas of the proof are as follows. First we show that as long as
t < d0,thetth central moment of the distance distribution of the code equals thetth mo- ment of the hypergeometric random variable with the pmf fX(i) = wi
m−w w−i
/ mw, i= 0,1, . . . , w.This fact relies on simple properties of the Johnson association scheme. After that we use a condition similar tow > k(w−d/2) (see the proof of Prop. 3) as sufficient for identification and write a Chernov-type bound that it is violated, where the random variables are the indices of krandom columns of the matrix A. This gives the estimate of in the form of the tth moment of a sum of certain independent random variables which can be bounded above using classical inequalities such the Rosenthal inequalities or other similar estimates. The full development of these arguments is rather long; see [135].
Using this information together with Theorem 12, we can compute upper bounds for the false-positive probabilities obtained from Eq. (8.5) for the various fixed-weight BCH codeword matrix examples given in Fig. 8.1. The results are listed in Table 8.42. We can see that the bounds are small, especially for smallerk, even when we have`= 2. Compared to all previous results, this gives better estimates of .
Chapter 9
Multilabel classification with
group testing and codes
9.1
Introduction
In the multilabel classification problem, we are given a set of labeled training data
{(xi, yi)}ni=1, where xi ∈ Rp are the input features for each data instances and yi ∈
{0,1}dare vectors indicating the corresponding labels (classes) the data instances belong
to. The vector yi has a one in the jth coordinate if the ith data point belongs to
jth class. We wish to learn a mapping (prediction rule) between the features and the labels, such that, we can predict the class label vector y of a new data point x
correctly. Such multilabel classification problems occur in many domains such as text mining, computer vision, music, and bioinformatics, and modern applications involve large number of labels. Popular applications with many labels include image and video annotation [136], web page categorization [137], text and document categorization [138], and others [139]. In most of these applications, the label vectors yi are sparse (with
average sparsity of k d), i.e., each data point belongs to a few (average k out of
d) classes. The multiclass classification is an instance of the multilabel classification, where all data points belong to only one of the dclasses (k= 1).
The simple binary classification problem, whered= 2 andk= 1 is well-studied, and several efficient algorithms have been proposed in the literature. A natural approach used to solve the multiclass (d >2, k= 1) classification problem is to reduce the problem
into a set of binary classification problem, and then employ the efficient binary classifiers to solve the individual problems. Popular methods based on this approach are: one-vs- all, all-pairs, and the error-correcting output code (ECOC) [140] methods. In ECOC method, m-dimensional binary vectors (typically codewords from an error correcting code with m ≤d) are assigned to each class, and m binary classifiers are learned. For the jth classification, thejth coordinate of the corresponding codeword is used as the binary label for each class. In the modern applications, where dis typically very large, this approach is found to be very efficient due to the reduction of the class dimension.
Related Work: The idea of ECOC approach has been extended to the multilabel classification (MLC) problem. In the multiclass classification, using codewords for each class in ECOC is equivalent to multiplying the code matrix to the label vectors (since the label vectors are basis vectors). In the multilabel setting, the ddimensional label vectors are reduced tomdimensional by multiplying a random matrixA∈Rm×dto the
label vectory. This reduction method was analyzed from the compressed sensing point of view in [141], with the assumption of output sparsity, i.e., y is sparse (with average sparsity k). Using compressed sensing (CS) theory, the results in [141] show that for a linear hypothesis class and under the squared loss, a random embedding (random code matrix) of the classes to m = O(klogd) dimensions does not increase the L2 risk of the classifier. However, the CS approach requires solving an optimization problem to recover the label vector. Constructions with faster recovery algorithms exist but we cannot obtain L2 norm results with them.
Alternatively, embedding based approaches have been proposed to reduce the effec- tive number of labels. These methods reduce the label dimension by projecting label vectors onto a low dimensional space, based on the assumption that the label matrix
Y = [y1, . . . , yn] is low-rank. The various embedding methods proposed in the literature
mainly differ in the way this reduction is achieved. The reduction is achieved using SVD in [107], while column subset selection is used in [108]. These embedding methods capture the label correlation, and Euclidean distance error guarantees are established. However, the low rank assumption breaks down in many situations [139], e.g., data is power law distributed [142].
135 Extreme Classification) [139] overcomes the limitations of previous embedding methods by first clustering the data into smaller regions, and then performs local embeddings of label vectors by preserving distances to nearest label vectors. However, this method also has many shortcomings, see [142]. Moreover, most of these embedding based methods are very expensive. They involve eigenvalue or singular value decompositions and matrix inversions, and may require solving convex optimization problems, all of which become impractical for very large d. In all the embedding methods and the CS method, the reduced label space is a real space (no longer binary). Hence we need to use regressors for training and cannot leverage the efficient binary classifiers for effective training for the model. Prediction will also involve rounding/thresholding of real values. This is additional work, and choosing a right threshold is sometimes problematic.
Contributions: In this chapter, we present a novel reduction approach to solve the MLC problem. Our approach assumes output sparsity (sparse label vectors withkd) similar to the CS approach, but reduces a large binary label vector to abinary vector of smaller size. Since the reduced label vectors are binary, we can use the efficient binary classifiers for effective training for the model. Our prediction algorithm is extremely simple and does not involve any matrix inversion or solving optimization algorithm. The prediction algorithm can also detect and correct errors.
We make the crucial observation that, the MLC problem can be solved using the group testing (GT) premise. That is, the problem of estimating the (few) classes of a data instance from a large set of classes, is similar to identifying a small set of items from a large set. We consider a group testing binary matrix A and reduce the label vectors
yi’s to smaller binary vectors zi using the boolean OR operationzi =A∨yi (described
later). We can now use binary classifiers on zi for training. Them classifiers learn to
test whether the data belongs to a group (of labels) or not. During prediction, the label vector can be recovered from the predictions of the classifiers using a simple inexpensive algorithm (requiring no matrix inversion or solving optimization algorithms). A low prediction cost is extremely desirable in real time applications.
The idea of grouping the labels helps overcome the issues most existing methods encounter; e.g., when the data has power law distribution [142], that is many labels have very few training instances (which is the case in most popular datasets), and tail
labels. Since the classifiers in our approach learn to test for groups of labels, we will have more training instances per group yielding effective classifiers. It is well known that the one-vs-rest is a highly effective method (expensive), and recently a (doubly) parallelized version of this method called DiSMEC [142] was shown to be very effective. Our approach is similar to one-vs-rest, but the classifiers test for a group of labels, and we require very few classifiers (O(logd) instead of d).
We establish Hamming loss error bounds for the proposed approach. Due to the error correcting capabilities of the algorithm, even if a fraction of classifiers mis-classify, we can achieve zero prediction error. Numerical experiments with various datasets illustrate the superior performance of our group testing approach with different GT matrices. Our method is extremely inexpensive compared to the CS approach and especially compared to the embedding based methods, making it very desirable for real time applications too.