Binding pure C libraries to Python is not very dicult, and the skills can be mastered in a rather short time frame. If done right, these bindings can be quite robust even towards certain changes in the evolving C API without the need of very time consuming manual tracking of all changes. As with many projects for this, it is vital to be able to automate the mechanical processes: Beyond the outlined code generation above, an important role
A.6. CONCLUSIONS ON PYTHON/NATIVE CODE INTEGRATION 177 comes to automated code integrity testing (here: using PyUnit [90]) as well as an API documentation (here: using Epydoc [91]).
Unfortunately, as CtypesLib is still work in progress, the whole process did not go as smoothly as described here. It was particularly important to match up working versions properly between GCCXML (which in itself is still in development) and CtypesLib. In this case a current GCCXML in version 0.9.0 (as available in Ubuntu Intrepid Ibex, 8.10) required a branch of CtypesLib that needed to be checked out through the developer's Subversion repository. Furthermore, it was necessary to develop a x for the code generator as it failed to generate code for#defined oating point constants. The patch has been reported to the author and is now in the source code repository. Also patching into the generated source code for overriding some features and manipulating the library loading code can be considered as being less than elegant.
Library wrapping as described in here was performed on version 1.16 of the LittleCMS library. While writing this, the author of LCMS has moved to the next stable version 1.17. Adapting the Python wrapper to this code base was a matter of about 15 minutes of work. The main task was xing some unit tests due to rounding dierences resulting from an improved numerical model within the library. The author of LittleCMS made a rst preview of the upcoming version 2.0 (an almost complete rewrite) available recently. Adapting to that version took only about a good day of modications, even though some substantial changes were made to the API. But even for this case only very little amounts of new code had to be written.
Overall, it is foreseeable that this type of library wrapping in the Python world will become more and more ubiquitous, as the tools for it mature. But already at the present time one does not have to fear the process. The time spent initially setting up the environment will be easily saved over all projects phases and iterations. Unfortunately we will probably not see Ctypes evolve to a point where it is able to interface to C++ libraries as well. The developers of Ctypes and Py++ (Thomas Heller and Roman Yakovenko) have evaluated potential extensions, but the complex intricacies of C++name mangling, and the dierences within the dierent compiler implementations, have proven to be too dicult for this. For now, we will have to accept the fact that it will not be possible to interface C++ libraries without an intermediate step of generating glue code that will be compiled to a binding library.
Appendix B
Applications in Industry Projects
During the research phase of the thesis, work to several industry projects was contributed. This appendix outlines briey work done for these projects. These were a baggage identi- cation for airport baggage management systems (Sect.B.1), a service oriented architecture based farm management system (Sect.B.2) and a quality control system for electronic con- trol consoles involving LCD displays (Sect.B.3).B.1 Baggage Identication
Baggage handling systems at airports are working during the rush hours at airports close to the limitations of their capacity. Certain errors in baggage identication do happen over the course a bag takes on the extensive conveyor belt systems. Each misidentied item has to recirculate the system. Due to the nearly complete utilisation of the system already, each of these faults therefore put an additional load onto the system, which is (due to tail backs) over-proportional to the extra time for the item's handling.
The idea of this work was to improve the accuracy of the baggage identication to reduce the ratio of identication failures. Currently, most baggage handling systems perform a complete tag based identication at the time the baggage item enters the main conveyor belt system using automatic bar code readers. At that point of time the bag is timed using a photo eye. Due to knowledge of belt speeds and distances the bag is identied at further decision points through timing using further photo eyes and knowledge of the bag order on the belt. Upon mis-measurement of only about 30 mm, a baggage item is agged as not correctly identiable and has to undergo a complete tag reading and security scan (X-ray) again. Such dierences easily occur if a bag's handle ips over or the bag shifts slightly in a turn.
Our approach was to build comparatively cheap smart cameras as appliances, that were to be distributed to the decision points, to be mounted above the conveyor belt. The
smart cameras contain an embedded computer capable of processing the live image stream and adapting to the current illumination condition. These cameras would observe the belt underneath and compute a set of specic metrics (digital ngerprints) from the images (see Sect. B.1.2). The metrics for each bag were to be sent o to a central matching server comparing the ngerprint to those stored in its database to identify a baggage item (see Sect. B.1.3). To compensate for dierences in light quality at dierent places of the belt as well as changing light conditions, the image appearance is rst (colour) corrected and normalised (see Sect. B.1.1). Colour correction for this case is performed in a similar fashion as outlined in this thesis: A foreground/background segmentation is performed, so that static background information can be extracted. Additionally, information from the colour histogramming is fed back to provide information on the change in colour appearance from a before state to a now state.
widen search sub set
determine key features send colour histogram adapt ICC profile determine colour histogram size/shape determination
match finger print against probable
sub set define search sub set
colour histograms motion unblur
[sufficient match]
[insufficient match]
[slight metric difference]
update database
[strong metric difference]
replace stored fingerprint update stored fingerprint send previously matching colour histograms Fingerprint Matching normalise image colour determine background mask send background mask Fingerprint Determination receive background mask receive colour histograms background mask receive colour histograms previous colour histograms Normalisation
& Colour Adaption
Figure B.1: UML activity diagram of the combined normalisation, visual ngerprinting and matching algorithm (including adaptive colour perception calibration and ngerprint updating) for the proposed system.
Fig.B.1outlines the processing ow within the proposed system. The two steps on the left hand side (normalisation/colour adaptation and ngerprint determination) operate on the smart camera. The ngerprint matching step is to be performed on a central system which communicates with the distributed camera systems. It receives the digital ngerprints and matches them to the ones stored in its database. It also feeds back colour histogram
B.1. BAGGAGE IDENTIFICATION 181 information of properly identied bags to the cameras to aid them in the adaptation process of the ICC prole (see Chap.8 and9).
A prototype of the system has been developed as a distributed system of software com- ponents for desktop computers. These components are described in the following sections. Beyond the three functions from Fig.B.1also a process control is present.
B.1.1 Image Normalization
Images acquired by cameras in distributed locations within the system are deteriorated due to various inuences. These inuences are:
• motion (resulting in motion blur),
• background (dierent visual characteristics of the baggage belt and surrounding envi- ronment in dierent locations),
• varying illumination conditions (natural, articial, changed due to the time of the day and/or external weather conditions, etc.) and
• dierences in image capturing devices' perception (dierences in the cameras due to model variation, batch production variation and ageing eects).
The background mask can also be used for the ICC prole adaption, as it separates the non-changeable fragments of the image from the changing fractions (bag).
In the prototype, the main application contains of a Python script, which calls into an image processing function in a native C/C++library for the benet of speed with the limited resources in an embedded device. This function performs motion un-blurring, an ICC prole colour correction and a statistical background/foreground segmentation. An adaptation of the ICC proles is currently still missing from the prototype. Communication between the distributed components is implemented through a service oriented architecture using Web Services.
B.1.2 Determination of Visual Fingerprint
On the basis of the normalised image the visual metrics composing the visual ngerprint can be determined. A set of easily determinable and robust properties useful for object identication are determined. Usable for ngerprints are for example these:
• Colour Histograms (hue histogram and saturation/chroma histogram) • basic size and shape (bounding box)
• pattern structure (extremes as solid colour or ne detailed regular pattern, can be determined analysing the frequency distribution after a Discrete Fourier Transforma- tion)
The ngerprinting code operates on the foreground areas of the normalised image rep- resentation as provided by the processing step outlined in Sect. B.1.1. The ngerprint contains the metrics relevant for object identication. This step also decides on a heuristic which of the captured frames of an individual item is ideal for matching. A ngerprint derived from that frame then is passed on to the matching server.
Currently, the ngerprint in the prototype contains a logarithmically scaled a∗b∗ chro- maticity histogram and a bounding box (size, aspect ratio and angle). It is returned from the native function called upon processing a suitable baggage image frame. The ngerprint data is communicated to the matching server via a Web Service call.
B.1.3 Matching Algorithm
The bags are normally moved on the belt in an undisturbed order. Thus, the corresponding ngerprint in the sequence of recorded ngerprints within the system is known. For matching the ngerprint of a given bag against the recorded ngerprints for identication, only a sub set of records around the approximated location will need to be performed. The match will be positive for a match with a high certainty in the proximity of signicantly lower correlation certainties.
In the case of an insuciently distinct positive correlation, the algorithm extends the search domain to locate the ngerprint within a wider range, if the bags' order got disturbed on the conveyor belt. The matching algorithm must provide sucient capabilities to with- stand these errors. The reason is on the one hand the nature of inevitable margins of error of measurements extracted from the image analysis, and on the other hand the fact that details of bags may look dierent in dierent locations (e. g. due to a moved bag handle).
To further reduce the potential margin of error in the ngerprint matching, the nger- print associated with a piece of baggage can be updated. In case of slight changes in the ngerprint, metrics should be updated through averaging of the metrics to decrease the sta- tistical variation. In case of signicant changes in metrics (e. g. due to a moved bag handle) the ngerprint would be replaced by the ngerprint of the current perception of the bag.
The information from a previous perception of an item at another location can be used to derive hints for a corrective adaptation of the ICC prole for the respective currently used camera.
In the prototype, the server matches the ngerprint against the database of stored n- gerprints, and the result is written to a log le. For the ngerprint matching, currently only the chromaticity histogram is used by performing colour indexing as described by Swain
B.2. MOBILE FARM MANAGEMENT 183