eNTERFACE’09 Project Proposal
Video Navigation Tool: Application to
browsing a database of dancers’
performances.
Principal Investigator: Xavier Siebert (FPMs, Mons)
1
Keywords
Multimedia database; Content-based video browsing; Dance performances
2
Project objectives
This project aims at creating an online navigation tool for video databases browsing. To allow for efficient browsing, the notion of "distance" between the items of the database must be defined, based on appropriate content descriptors. As part of the Numediart program (www.numediart.org, in particular the {Audio, Media}Cycle projects) we are developing content de-scriptors for sound (e.g., rhythm, harmony, timbre, . . . ) and images (color, shape, textures, . . . ). The combination of sound and images paves the way for treating video databases, which nevertheless require additional specific features (e.g., quantity of motion).
This eNTERFACE09 project will focus on a particular type of videos, namely short (2 min) dancers performances, that have been filmed in a pre-cise setup. As such, it is associated with the artistic project "DANCERS!" launched by Bud Blumenthal’s collective (www.bud-hybrid.org), which aims at providing the first high quality audiovisual database of professional dancers
A major challenge of this project is to find descriptors of dancers’ move-ments that will provide relevant criteria to navigate the database. Some such descriptors are already incorporated into the EyesWeb [CRT99] XMI plat-form (e.g., quantity of motion, contraction index, . . . ), and others will need to be developed. Additional tags (name, age, city, body height, . . . ) will be associated with the videos, to provide a mixture between tag- and content-based browsing.
A website will be created to provide online access to the dancers’ infor-mation, and to interact with users voting by mobile phone.
3
Background information
The evergrowing amount of publicly available multimedia libraries calls for novel tools and techniques for accessing their content efficiently. Database search initially relied on metadata (e.g., captions, keywords) associated to each item (sound, image, video, . . . ) which may be difficult or tedious to generate, especially when the database reaches a critical size. Content-based descriptors have been developed as an alternative or a complement to metadata, as in the "Query by Image and Video Content" (QBIC) system [FSN+95]. However, finding appropriate content descriptors is not a trivial task, particularly in the case of videos [PJ02, LSDJ06].
Several algorithms for compression and recognition of dance gestures have been developed recently [eFC04] and could serve as a basis for providing de-scriptors to organize the database. Other criteria for gesture recognition and analysis will be developed a Numediart project, "Dancers Gestural Analysis by Dual Remote/Worn Sensing".
The associated artistic project "DANCERS!" consists of a performance installation, that will be moved from cities to towns, featuring a 6m-wide projection screen besides a stage on which professional dancers will perform. A high quality camera will record their performance and display it on the screen, supported by a surround sound system. Dancers will perform one at a time on stage during two minutes, and the recorded performance will be stored in a database along with their biometric data (name, age, city, body height, . . . ). The audience could be allowed to vote to rate the performance or to select the next dancer to perform, using a tabletop booth or even cellular phones.
4
Technical description
Below are listed specific points of technical relevance that will be investigated during the eNTERFACE workshop.
• given the recordings of two video cameras (one above and one in front of the stage), dancers movements have to be tracked in 3D and mean-ingful analytical descriptors of their gestures have to be extracted. To allow for comparison of whole sequences, global features have to be defined first : mean and variance of space occupation, statistics on the dynamics of the motion (e.g., quantity of motion), . . . The relevance of local features has to be investigated as well.
• organize and navigate the database containing audiovisual records, user-submitted tags and dancers’ biometric information. Several algo-rithms based on graph theory can be incorporated to efficiently retrieve similar dancer’s gestures.
• develop social network / web 2.0 functionalities for the website • improve the usability of the tabletop booth
5
Resources needed
The resources listed below are necessary to process video sequences that will be provided by Bud Blumenthal.
• Computer resources: Development will be achieved on participant?s laptops, and the softwares are all publicly available. Some parts of the project will rely on the EyesWeb XMI or on the tools developed as part of the Numediart project. An online server will be setup to store the database of audiovisual rushes of dancers performances and to host the website with the navigation tool.
• Staff: 1 project leader + 5 researchers.
• Additional resources: a tabletop booth for public votes, to allow rating of the performances.
6
Project management
The project leader will be responsible for the management, accompanied by a researcher from his team.
6.1
Work plan and implementation schedule
The recording of dancers’ performances will take place before the eNTER-FACE workshop. The first video tests were done on January 28 and 29, 2009, with the participation of the principal investigator of this project. Other video recordings will take place between February and June, so that a small database of videos will be at out disposal at the beginning of the eNTER-FACE workshop, which we are planning to organize as follows:
1. First week
• Kickoff meeting • Work assignment
• use of EyesWeb XMI to track the dancer’s motions and extract representative features (space occupation, quantity of motion, con-traction index, . . . ) (to start)
• design and develop the website (to start) 2. Second week
• continue the development of descriptors to classify the dance videos. Eventually use EyesWeb’s Software Development Kit (SDK) to contribute new blocks specific to the analysis of dancers’ motion. • design and develop the website (to continue)
3. Third week
• design and develop the website (to continue)
• incorporate the descriptors of dancers motion into the website • prototype of user interface for the voting tabletop
• Demonstrate the prototype of the online navigation tool and the voting tabletop
• Paperwork • Review meeting
After the eNTERFACE workshop, a public presentation of the related performance with public voting is scheduled for November 2009.
6.2
Deliverables
• Navigation software for video browsing
• Rich Internet Application of the web 2.0 application for browsing the database
• Prototype of user interface for the voting tabletop
6.3
Benefits of the research
The final product of this research will be a website for navigating a database of dancers. The mixing of tag- and content-based browsing opens the door to other studies, and will be an interesting tool for choreographers.
6.4
Profile of team
6.5
Leader
Xavier Siebert graduated as a physics engineer from ULB, Belgium and holds a Ph.D. in biophysics from the Johns Hopkins University (Baltimore, USA). His thesis work focused on the determination of the three-dimensional struc-ture of proteins and on 3D modeling. He then moved to France to work as a post-doctoral researcher at the CNRS on software development for in-terfacing X-Ray crystallography and Electron Microscopy. His interest is in applied mathematics and digital image processing.
He now works as a Senior researcher at the Polytechnic Institute of Mons (aka FPMs), as part of the Numediart project. He is the co-director (along with Stéphane Dupont from FPMs) of the MediaCycle project, which inves-tigates software tools for browsing sounds, images and their future extensions
to video databases. The first prototype of the software for audio browsing has been presented recently at Multitel, Belgium, and is currently being adapted to other types of media.
6.6
Staff
• Staff proposed by the leader: one researcher from the Numediart team will join the project. Possible candidates include Stéphane Dupont (senior researcher, co-director of the MediaCycle project) and Sullivan Hidot (post-doctoral researcher who worked on the statistical analysis of dancers movements). The availability of the researcher from the Numediart team will depend on the acceptance of other eNTERFACE projects, and cannot be further specified at this stage.
• Other researchers needed:
– 2 researchers working on video analysis
– 2 researchers working on website development
– 1 researcher working on user interface for the voting tabletop A possible industrial partner for the design of the website is Speculoos, based in Belgium, member of the Open Source Publishing agency
6.7
Artistic collaborations
• Bud Blumenthal’s collective (along with Charleroi-danse) will provide the recordings of dancers performances.
References
[CRT99] Antonio Camurri, Matteo Ricchetti, and Riccardo Trocca. Eye-sweb - toward gesture and affect recognition in dance/music inter-active systems. Multimedia Computing and Systems, International Conference on, 1:9643, 1999.
[eFC04] S. Boukir et F. Chenevière. Compression and recognition of dance gestures using a deformable model. Pattern Analysis and Appli-cations, Springer-Verlag, 7(3):308–316–, Décembre 2004.
[FSN+95] Myron Flickner, Harpreet Sawhney, Wayne Niblack, Jonathan Ash
ley, Qian Huang, Byron Dom, Monika Gorkani, Jim Hafner, Denis Lee, Dragutin Petkovic, David Steele, and Peter Yanker. Query by image and video content: The QBIC system. Computer, 28(9):23– 32, 1995.
[LSDJ06] Michael S. Lew, Nicu Sebe, Chabane Djeraba, and Ramesh Jain. Content-based multimedia information retrieval: State of the art and challenges. ACM Trans. Multimedia Comput. Commun. Appl., 2(1):1–19, February 2006.
[PJ02] Milan Petkovic and Willem Jonker. Cobra: A Content-Based Video Retrieval System. 2002.
