With the tetrahedral catalogue, line labelling can in principle be used with equal success for drawings of trihedral and tetrahedral objects. However, the dramatic increase in the number of valid labellings noted in the previous section presents practical problems. Since it was the sparsity of the junction catalogue which led to practical low-order timings in the trihedral case, and since the tetrahedral catalogue is no longer sparse (at least in the cases of L-, W - and Y -junctions), exponential rather than low-order polynomial behaviour is observed in practice: for example, Figure B.503 has more than 200000 valid labellings, and evaluating all of them is clearly impractical.
The approach in this thesis starts with one advantage: since, by assumption, a drawing shows a single, entire object, all outer boundary lines in the drawing must be occluding. This assumption is not valid for applications such as identification of buildings from aerial photographs: in such cases, the potential for ambiguous labellings is greater, and thus the labelling problem would be even harder.
Junction Label Vertex Types Xabcd, Xbcda, Xcdab, Xdabc Mccdc Xabdc, Xbdca, Xdcab, Xcabd Mcdcc Xabdd, Xbdda, Xddab, Xdabd Mcddc
Xcccc Xcccc
Xcccd, Xccdc, Xcdcc, Xdccc Xcccd
Xcdcd, Xdcdc Xcdcd
Xcddd, Xdddc, Xddcd, Xdcdd Xcddd
Xdddd Xdddd
Zcbda Zcdcdcd
Xccccc Xccccc
Xddddd Xddddd
Xcccccc Xcccccc
Xdddddd Xdddddd
Zcdcdcd, Zdcdcdc Zcdcdcd
Table 4.8: X-type and Z-type Junction Labels
Furthermore, whereas in the trihedral case, each junction label determines un-ambiguously the underlying vertex type of the corresponding vertex, this does not extend to the non-trihedral case. In the worst case, a single junction label can be in-terpreted as any of seven underlying vertex types: two convex and one concave edges;
three convex and one concave edges, in K-, M - or X-configuration; two convex and two concave edges, in K-, M - or X-configuration [175]. The pairs of drawings in Figure 4.34 show three of these interpretations (in each pair, the left-hand drawing shows the ambiguous junction label and the right-hand drawing shows the revealed vertex type).
Figure 4.34: Three objects illustrating different interpretations of a single junction label [175]
A deterministic algorithm [64] which has been used successfully to label trihedral drawings [172] has been adapted for the case where multiple valid labellings are the norm, not the exception. As expected, there are drawings for which its time performance is unacceptable.
Junction Label Vertex Types
Yaab, Yaba, Ybaa Mccdc
Yabb, Ybba, Ybab Mcdcc
Yabc, Ybca, Ycab Xcccd
Yabd, Ybda, Ydab Wcdc, Xcccd, Mccdc, Mcdcc, Kcccd, Kccdc, Xcdcd, Mcdcd, Mdcdc, Kcdcd, Kdcdc
Yacc, Ycca, Ycac Xcccd
Yacd, Ycda, Ydac Xcdcd, Mcdcd, Kdcdc Yadd, Ydda, Ydad Xcddd, Mddcd, Kdcdd Ybdc, Ydcb, Ycbd Xcdcd, Mdcdc, Kcdcd Ybdd, Yddb, Ydbd Xcddd, Mdcdd, Kddcd
Yccc Yccc
Yddd Yddd
Table 4.9: Y -type Junction Labels
Junction Tri- Extended Tetra- Total Type hedral Trihedral hedral Valid
L 6 0 2 8/16
T 4 4 12 20/64
W 3 0 25 28/64
Y 5 0 27 32/64
K 0 0 8 8/256
M 0 0 11 11/256
X 0 1 24 25/256
Table 4.10: Number of Entries in Junction Catalogues
The problem can be considered as a search problem, where the space to be searched is the set of valid labellings, and the search criterion is a heuristic measure of the merit of a particular labelling. Non-deterministic algorithms have been used with some success in NP-complete search problems. To investigate this possibility, a non-deterministic algorithm has also been tested to see if it improves on the deterministic algorithm.
The non-deterministic algorithm chosen is based on probabilistic relaxation la-belling [140]. To avoid confusion, the deterministic method, sometimes called “dis-crete relaxation labelling”, is called set-intersection labelling in this thesis.
4.4.1 Heuristics
There are two distinct purposes for which heuristics can be used in improving the performance of any search: choice heuristics assist in choosing the best interpretation when there are many valid interpretations, and pruning heuristics help to speed up the search by lopping off unlikely branches of the search tree.
Heuristics used for line labelling can also be subdivided into global heuristics, based on measures derived from the drawing as a whole, feature heuristics, measures derived from a part of the drawing, and local heuristics, measures based on individual lines or junctions.
RIBALD assigns a figure of merit to each labelling, and the labelling with the highest figure of merit is the one preferred. This overall figure of merit is the product of the figures generated by each heuristic.
Global Labelling Heuristics
It seems plausible that the object should be as “simple” as possible. If the object represented in a drawing has any self-similarity (repetitiveness or symmetry), the number of different underlying vertex types (the numbers and types of edges in-cident at the vertices corresponding to junctions) in the object will be small. One reasonable heuristic is that, as far as possible, the labelling should minimise the number of different vertex types.
Since some junction labels determine uniquely the underlying vertex type, while others do not, the merit figure is assessed in a two-stage process. Firstly, those junction labellings which correspond unambiguously to a single underlying vertex type are noted, and the minimum set of underlying vertex types established. The final count is then the sum of the number of underlying types in this set and the number of ambiguous junction labellings which cannot be interpreted as any of the labellings in this set (in practice, the second number is almost always zero).
Numerically, this preference is quantified as (Lu+Ls)−kL, where Lsis the number of different underlying vertex types required by those junction labels which corres-pond to unique underlying vertex types, Lu is the number of junction labels which cannot be interpreted as one of the unique underlying vertex types already required and counted in Ls, and the sum (Lu + Ls) provides an estimate of the number of
distinct underlying vertex types in the object; kL is a tuning constant.
In order to identify an optimum value of the tuning constant kL (and other tuning constants described below) these were input as parameters to a downhill optimisation process where the objective function being minimised was the number of incorrectly-labelled edges in my set of test drawings (see Appendix C). Results suggest that the optimum value of kL is between 0.5 and 0.6, implying that this heuristic is reasonably useful in identifying preferred labellings.
It is also plausible that if a drawing is of a single object, it should be as “connec-ted” as possible1—occluding lines in the interior of the sketch could be considered undesirable (for example, in Figure 4.35 on page 78, the line marked * should not be occluding). This is modelled by assigning a figure of merit (1− EEot)kE to the la-belling, where Eo is the number of occluding lines, Et the total number of lines, and kE a tuning constant. However, investigations suggest that the optimum value of kE is close to zero, implying that this heuristic is of little use in identifying preferred labellings.
Feature Labelling Heuristics
RIBALD tests for the presence of certain features—pockets, bosses and slots—as described in Chapter 6. Each such hypothesised feature has an associated figure of merit Mh and requires the lines forming part of that feature to be labelled in a particular way.
RIBALD calculates a feature merit figure for a labelling: the product of the merit figures Mc for each feature, where Mc = 1 for labellings which match the expectations for the hypothesised feature, and Mc = (1− Mh)Kc for labellings which do not match the hypothesis (Kc is another tuning constant).
Local Labelling Heuristics
Some junction labels are “better”—more common, or more plausible—than others.
Preliminary investigations show that even simple heuristics such as “interpret as many T -, W - and Y -junctions as trihedral as possible” identify the favoured inter-pretation in about half of the cases tested, including those shown in Figures 4.27
1Mackworth’s POLY [96] is an extreme example of this, generating all possible interpretations of a drawing in decreasing order of connectedness
and 4.28 on page 68, and in most other cases left the favoured interpretation in the first few when arranged in a preference order based on this heuristic.
RIBALD uses a more sophisticated model, assigning a constant figure of merit to each junction label (determined using the optimisation process described above and normalised so that the “best” label for any junction type has a figure of merit of 1.0). The contribution of this heuristic to the overall figure of merit for the labelling is the product of the figures of merit for each individual junction label.
4.4.2 Deterministic Labelling
As a particular deterministic labelling method, RIBALD implements a set-intersection labelling approach derived from Kanatani’s iterative constraint propagation
method [64]. At its core is the following algorithm:
• (Initialisation)
• For each junction, candidate label set = all valid labels for that junction type;
• For each boundary line, candidate label set = {occluding such that outside is occluded}
• For each non-boundary line, candidate label set = {occluding to left, occluding to right, convex, concave}
• Set of junctions to be processed Sj = {all junctions};
• Set of edges to be processed Sl ={all edges};
• (Processing)
• Loop
• – For each junction in Sj
∗ Eliminate from the candidate label sets for neighbouring lines any line labels inconsistent with the remaining candidate labels for this junction;
∗ If the junction label is unique, remove the junction from Sj; (see Note 2 on Page 78)
– For each line in Sl
∗ Eliminate from the candidate label sets for the neighbouring junctions any junction labels inconsistent with the remaining candidate labels for this line;
∗ If the line label is unique, remove the line from Sl
– Exit the loop if Sj and Sl are both empty (a unique labelling has been obtained)
– Exit the loop if the set of candidate labels for any junction or line is empty (no valid labelling can be obtained given the starting conditions)
– Exit the loop if no candidate labels were eliminated in this iteration
• End Loop
This algorithm is demonstrably O(n2) (with n being the number of lines in the drawing). In the worst case, each iteration of the loop removes a single candidate junction or line label; the number of these is proportional to the number of lines.
In practice, the algorithm is sufficient to obtain a unique labelling for the majority of drawings which meet the simplifying assumptions that the object drawn is a trihedral polyhedron with no through holes or hole loops (Parodi [120] reports the same result with a different deterministic algorithm and a far larger set of test drawings). In cases where these assumptions do not hold, it is likely that there will be several valid labellings, and further processing required to identify the preferred one.
The algorithm has three exit conditions: no valid labelling, a unique labelling, or no further progress. In the first two cases, it need only be called once. In the last case, it requires a surrounding control structure. At least one junction and at least one line still have multiple possible labels. It is likely that there will be multiple valid labellings compatible with the starting conditions (although this is not certain, as one or more of the remaining possible junction or line labels may be a “dead end”).
To allow for ambiguities, the following control structure is added:
• If the algorithm terminated ambiguously
– Choose any junction or line where the set of candidate labels L contains more than one element; (see Note 1 later)
– Choose any element l of this set;
– Create a labelling A which is identical to the original labelling except that the label set for the chosen junction or line is {l}
– Label the rest of A by reentering the algorithm at (Processing) above;
– Create a labelling B which is identical to the original labelling except that the label set for the chosen junction or line is L− {l}
– Label the rest of B by reentering the algorithm at (Processing) above;
• End If
In practice, this extension to the original algorithm is adequate when the number of alternative labellings is small, as is commonly the case with drawings of trihedral polyhedra with through holes or hole loops. However, it appears that the worst case (where each disambiguation step fails to propagate to neighbouring junctions or lines) is O(4n) and this pessimistic prediction is nearer the truth if the non-trihedral junction catalogue is used. If it were possible to identify in advance which apparently-trihedral junctions corresponded to trihedral vertices and which to tet-rahedral vertices, so that the full non-trihedral catalogue is used only for the latter, the problem would be alleviated considerably. This does not seem to be possible.
For example, it might be possible to infer on the basis of symmetry that T -junction A in in Figure 1.1 (page 2) is a reflected K-junction, but deducing that the object is symmetrical before obtaining the labelling is hard (and not all symmetrically-related junctions are as close to one another as this pair).
In an attempt to speed up practical labelling, RIBALD (a) tries to search the most promising branches of the tree of valid labellings first, and (b) tries to lop off unpromising branches of the tree without processing them.
Note 1: In order to search the most promising branches of the tree first, the choice of which junction or line to disambiguate, and which candidate label to choose to investigate first, is made according to a priority list. Prior to labelling, a list of the
“most desirable” junction labels is generated; this is based both on fixed priorities for the most common junction labels and on the merits of candidate features (Chapter 6)
and the resulting junction labels they imply. When the control structure requires a choice, this list is searched, and the highest-priority junction label which would produce a disambiguation is chosen.
Note 2: In order to lop off unpromising branches of the tree, the local contribution to the labelling merit is calculated as each unambiguous junction label is identified.
If the current labelling merit of the branch under investigation is already seriously worse than the merit of the best labelling so far, the branch is lopped off. (“Seriously worse” is implemented as Ln< 2L0− 1, Ln being the current labelling merit and L0 the best so far; if a single “best” labelling is wanted rather than a choice of reasonably good ones, performance could be improved further by using Ln < L0 instead.) (Note that slightly-unpromising branches will not be lopped off: Ln is based solely on the local contributions to the labelling merit, whereas L0 also includes the global and feature measures.)
Even this approach can be unacceptably slow for interactive response times, so RIBALD forces the algorithm to return the best labelling it can find in a specified time by limiting the number of tree nodes examined. In any branch of the tree, at most N labellings are examined; these are subdivided whenever the tree branches again, so that the most promising branch is allocated pN nodes and the remaining alternatives (1− p)N nodes. (The results in Section 4.5 were obtained using p = 0.7 and an initial N = 2000.)
*
Figure 4.35: One Ob-ject or Two? [194]
Figure 4.36: The Tetrahedral Catalogue must be used to label these drawings correctly
For several test drawings there is a valid labelling using a more restrictive junc-tion catalogue, but a better labelling can be obtained using a less restrictive cata-logue. For example, Figure 4.35 can be labelled using the trihedral catalogue (as two unconnected objects), but the labelling obtained using the extended trihedral
catalogue is clearly preferable. Similarly, the drawings in Figure 4.36 have valid labellings using the extended trihedral catalogue, but a clearly superior labelling can be obtained using the tetrahedral catalogue. While it is possible to create heuristics to choose the non-trihedral interpretation once it has been generated, it seems impossible to know, without generating non-trihedral interpretations, that the trihedral interpretation is not the best.
However, it is also observed that the less restrictive junction catalogue may result in an inferior labelling when the superior labelling can be obtained using the more restrictive catalogue—for example, the superior labelling may be in an unpromising branch of the tree which has been lopped off. It is not intuitively obvious which is the more common occurrence, so for purposes of comparison, RIBALD implements various options for the junction catalogue (the labels in brackets refer to rows of Table 4.13 on page 86):
• (SI-Full) use the full catalogue (all trihedral, extended trihedral and tetra-hedral junction labels plus the common symmetrical 5-tetra-hedral and 6-tetra-hedral labels);
• (SI-LWY) use the trihedral catalogue for L-, W - and Y -junctions, and the full catalogue for other junction types; if no valid labelling is obtained, use the full catalogue instead (several drawings, including Figures 1.1 and 1.2, can be labelled using this method);
• (SI-X3h) use the extended trihedral catalogue; if no valid labelling is obtained, use the full catalogue instead;
• (SI-3h) use the trihedral catalogue; if no valid labelling is obtained, try again using the extended trihedral catalogue and if necessary the full catalogue.
If non-trihedral junctions occur in drawings of plausible engineering objects, rather than in the simple illustrative solids shown here, some method of localising the non-trihedrality would prevent the generation of large numbers of implausible labellings which will inevitably be discarded. Ideally, the initial set of valid labellings should be the trihedral set only for those junctions “close” to the centre of non-trihedrality; elsewhere, the trihedral set should be used. My initial investigations
showed that defining “close” as one or two edges away from a visibly non-trihedral vertex was inadequate (for example, it takes no account of object symmetry), and this idea has not been pursued.
4.4.3 Probabilistic Labelling
As a particular non-deterministic labelling method, RIBALD implements a relaxa-tion labelling algorithm. Relaxarelaxa-tion methods have been used successfully in several machine vision processes from scene labelling [140] to object recognition in robotic systems [136], and probabilistic relaxation can be viewed as a natural extension of constraint propagation, so probabilistic relaxation was therefore my first choice.
Alternative approaches are possible. Genetic algorithms might be worth revis-iting. The disappointing results in [153] were obtained several years ago. More recently, Myers [114] reports successful results in labelling trihedral scenes with genetic algorithms; he stresses the advantage of producing a population of valid la-bellings, rather than a unique labelling. He gives no timings but indicates that the order of the algorithm is the same as the order of the fitness function, i.e. polyno-mial. Ant systems [22] also appear worthy of investigation, although it is not clear at this point whether an implementation would differ significantly from probabil-istic relaxation labelling, and if the analogy with crystallisation used below is valid, simulated annealing [67, 108] might also be a method worth investigating.
The algorithm as implemented is as follows:
• (Initialise)
• For each junction, allocate a probability (see text) for each candidate junction label for junctions of that type, such that each probability is greater than 0 and the sum of all probabilities at the junction is 1.
• For each boundary edge, set the probability that the edge occludes the outside to 1 and the probabilities that the edge occludes the inside, is convex, or is concave, to 0.
• For each non-boundary edge, allocate a probability for each candidate label (oc-cluding to left, oc(oc-cluding to right, convex or concave), such that each probability
is greater than 0 and the sum of all probabilities is 1.
• Set of junctions to be processed Sj = {all junctions};
• Set of junctions to be processed Sj = {all junctions};