There exists different components to proposed PTM datamining framework. In proposed PTM datamining we have focused on the components relevant to pattern discovery techniques like sequence pattern mining, clustering and classification. Particularly we are focusing on the required
PTM datamining framework
Multimedia data Reporting & using:
Feature extraction: Representation: Transformation: Pattern discovery: Interpretation:
Concept-ba sed nea r- duplica te video detection
for novelty re-ra nking Accura te frequent event
pa tterns for beha vior a na lysis in meetings Composite concept detections Shot- boundary detection Key-frame extraction
Edged direction histogram (EDH), Gabor (GBR), Grid color moment (GCM)
Columbia374 concept detectors
PTM
data
Sequence
Pattern
Mining
PTM
data
Clustering
PTM
data
Classifi-
cation
Figure 3.1: Proposed PTM datamining framework.
representations, transformations, pattern discovery and interpretation techniques for PTM datamin- ing. While components for low level feature extraction and primitive concept detections are adopted from existing methods as proposed in [139] and they are widely utilize for Text Retrieval Confer- ence Video (Trecvid) retrieval evaluation [95] challenges. In Figure 3.1, we will illustrate briefly each of the component of PTM datamining framework and details of proposed novel techniques for each of the components are described in corresponding MDM applications developed in chapters 4, 5 and 6.
Low level feature extraction and concept detectors: As first step of video processing we extract the keyframes from videos. If the videos considered are edited then we use Fraunhofer Institute and Dublin City University teams shot boundary detector and then consider middle frame of the shot as the keyframe. otherwise, for the unedited videos keyframes are extracted every ∆t seconds. Three low-level visual features: edged direction histogram (EDH), Gabor (GBR), and grid color moment (GCM) are extracted from each keyframes and Columbia374 trained SVM models for suitable concepts are applied as per the guideline in [151].
Representation:The representation stage involves integrating data from different sources and/or making choices about representing or coding certain data fields that serve as inputs to the
pattern discovery stage. This stage is of considerable importance in multimedia datamining. In PTM framework we proposed two novel representations PTM event sequence in chapter 4 and PTM time-series of concept confidence values in chapter 5 and 6.
Transformation:It is an important component as multimedia data are often the result of outputs from various kinds of sensor modalities with each modality needing sophisticated prepro- cessing, synchronization and transformation procedures. We have described proposed transforma- tion technique for such multimodal data in chapter 4. Also, transformation of PTM time-series to categorical data is described in chapter 5 to reduces the dimensionality and increases the scalability for unequal length videos. Whereas another transformation techniques were proposed in chapter 6 for robust knowledge discovery.
Pattern discovery: It is the component where the hidden patterns, relationships and trends in the data are actually discovered. As PTM data has novel representation it may not be possible for many existing algorithms to process such data directly. Thus, we proposed a novel pattern discovery algorithm like PIE-Miner for sequence pattern mining as in chapter 4. Also, we apply novel transformations techniques on proposed PTM time-series representation in chapter 5 and then applied existing pattern discovery algorithm like COBWEB for clustering.
Interpretation: To evaluate the quality of discovered pattern and its utility for proposed applications we proposed novel interpretation techniques or modified existing. As the discovered frequent event sequence patterns from PTM data are unique compared to existing sequence pattern mining methods we proposed novel interpretation with notion of strong and weak patterns and tau- containment in chapter 4. Whereas novel interpretation of concept-based near-duplicate videos for novelty re-ranking is done in chapter 5. Similarly, novel discovered Adaptive ontology rules are utilized for interpreting composite concepts in chapter 6.
Reporting and using discovered knowledge: Finally reporting and putting to use the discovered knowledge to generate new applications like semantic level novelty re-ranking in chapter 5. Also, the traditional behavior analysis applications or ontology rule based composite concepts detection application are expanded in terms of kind of knowledge discover from group meeting behavior analysis with novel patterns in chapter 4 and novel Adaptive ontology rule discovery in chapter 6.
Problem Definition of PTM datamining: Let S be a multimedia system designed for accomplishing a set of detection tasks Tr = {T1, T2, . . . , Tr}, r being the total number of de-
{M1, M2, ..., Mn} be the set of n correlated media streams. Let L = {l1, l2, . . . , lr} be the se-
mantic labels output by the various detectors Tr.
For 1 ≤ i ≤ n, let 0 < pMit
j < 1 be the probability of label l Mi
j output by the detector Tj based on individual ith media stream at time t. The time is represented by starting time and
ending time, representing the duration of symbol existence in the stream. pMit
j is determined by
first extracting the low level content features from media stream i and then by employing a detector (e.g. a trained classifier) on it for the task Tj. The dataset generated with such multimedia system is
called as ”probabilistic temporal multimodal dataset”. Thus, we obtain a set of n correlated labeled streams correspond to the n media streams: L ={L1, L2, . . . , Ln}
where L1={(lM1 1, p M1t 1 ), (l M1 2 , p M1t 2 ), . . . , (lrM1, pMr 1t)} L2={(lM1 2, p M2t 1 ), (l M2 2 , p M2t 2 ), . . . , (lrM2, pMr 2t)} Ln={(l1Mn, p Mnt 1 ), (l Mn 2 , p Mnt 2 ), . . . , (lrMn, pMr nt)}
In the following subsections we will look at multimodal datamining problems arising on our probabilistic temporal multimodal dataset.