Introduction to the
Perceptual Computing
Designed by employees of The Center of Innovative Education
Introduction to the Perceptual
Computing
This course is aimed to study and introduce the innovative technologies of human-computer interaction to the educational process in institutions of higher and secondary education.
The goals of the course:
• To study the fundamentals of human-computer interaction; • To get acquainted with the hardware and software features of
the Intel Perceptual Computing SDK;
• To study how to design the interactive applications, which involve functions of natural intuitive interaction.
Introduction to the Perceptual
Computing
• The course is designed in the collaboration with Intel Corporation
• The set of lectures and practical works can be taught as elective course or as distance course
Authors
• The leader of the group – Olga Yufryakova, Director of the Center of Innovative Education, Institute of mathematics, information and space technologies NaRFU, senior lecturer at the Department of programming and high performance computing.
• Julia Berezovkaya, senior lecturer at the Department of programming and high performance computing.
• Vera Nekrasova, an engineer at the Center of Innovative Education, Institute of mathematics, information and space technologies NaRFU, M.Sc. In Media Technologies.
• Konstantin Nosov, a software developer at the Center of Innovative Education, Institute of mathematics, information and space technologies NaRFU, M.Sc. In Media Technologies.
Actuality
• Natural intuitive interaction is the evolution of the interaction with the computer through natural for human ways and interfaces (beyond keyboard and mouse);
• All that you are trying to solve with the help of a computer, will be done through the recognition of your face, voice, gestures.
• Natural and intuitive interaction involves the use of various technologies, transforming the interaction with the computer in a rich and interesting process.
Course structure
Lectures Practical work
1 New features in human-machine interface. Basic terms, the benefits of utilization.
2 Fundamentals, functionality and features of Intel Perceptual Computing SDK.
Installation and configuration of Intel Perceptual Computing SDK.
3 Gesture recognition with Intel Perceptual Computing SDK.
Designing of applications for gesture recognition using SDK.
4 Voice recognition with Intel Perceptual Computing SDK.
Designing of applications for voice recognition using SDK.
5 Face detection and recognition (face landmarks and characteristics) with Intel Perceptual Computing SDK.
Designing of applications for face detection and recognition using SDK.
6 Tracking of 2D/3D objects, augmented reality. Designing of applications for tracking of 2D/3D objects using SDK.
7 Game engines and integration with Intel Perceptual Computing SDK. OpenFrameworks: concepts, features, functionality.
Configuring of SDK for use with openFrameworks applications. Installation of openFrameworks. An example of application.
Hardware (1)
CREATIVE Interactive Gesture Camera
•Sensor resolution: 720p (1280 x 720)
•Resolution of depth sensor: QVGA (320 x 240) •The angle of view: 73
•30 FPS
•Working distance: from 15 cm to 1 meter •Power supply: USB 2.0 (< 2.5 W)
Hardware (2)
Center of Innovative Education, Institute of mathematics, information and space technologies NaRFU is equipped by:
• 15 working places;
• Three CREATIVE Interactive Gesture Camera; • Installed software;
Software
Intel Corporation provides an innovative product “Intel Perceptual Computing SDK”.
SDK is focused on several aspects of natural intuitive human-computer interaction:
•Face Detection; •Voice recognition; •Gesture recognition.
Also SDK includes tutorials, code samples, algorithms, and application examples. This will allow developers to implement functions of natural intuitive interaction.
Modules of
Intel Perceptual Computing SDK
• Gesture recognition module; • Face detection module;
• Voice recognition module;
Gesture recognition module (1)
Detection of fingers position (7 points): •The tips of the fingers;
•Center of the palm;
•The most visible point, which is located close to the elbow.
Gesture recognition module (2)
Recognition of standard gestures: –Thumb up;
–Thumb down; –Victory.
–Open palm.
Recognition of dynamic gestures: –Swipe, circle, waving.
Face detection module (1)
The module allows to get a lot of information about the recognized face (7 points on the face):
•Corners of the eyes; •Corners of the mouth; •The tip of the nose.
Face detection module (2)
The module allows to get some analytical information about the recognized face:
•Age group of the person who presents in the video stream (a child, an elderly person);
•Gender; •Wink; •Smile.
Voice recognition module
The module offers the following functionality: –Realization of voice menus;
–Dictation;
–Speech synthesizes.
At the moment only English dictionary is available. Other languages will be added during co-operation with the developers of voice engine.
Now for the implementation of voice control voice engine Nuance Dragon Assistant is used. Russian language will be also available soon.
Module for tracking of 2D/3D objects
Functionality of this module allows to track 2D/3D objects as well as faces.
As the result will be obtained: • Position of the object;
• The angle of inclination, etc.
The module supports tracking of three-dimensional objects that are specified in the form of a model ( .obj file).
Working with game engines
SDK implements several levels of interfaces that allow the immediate use of game engines. At the moment the following game engines are supported:
• Unity;
• Processing;
Technology application (1)
Video games:
• Interaction with the three-dimensional world and other players;
• Utilization of voice or gestures to manipulate the game.
Technology application (2)
• Modeling of grabbing,
moving system of virtual 3D object;
Technology application (3)
Interaction with media content:
• For example, switching
tracks in the playlist or flipping through book pages.
Technology application (4)
Security systems:
• Simple person identification (face-login).
• The combination of face, voice and gesture
parameters for more precise identification.
Technology application (5)
Image segmentation:
For example, to add more colors to the virtual communication. The background can be changed by any image, such as the lunar landscape or ocean beach.
Links
Northern (Arctic) Federal University named after M. Lomonosov
http://www.narfu.ru
Intel Perceptual Computing SDK
http://www.intel.com/software/perceptual Demo application
http://software.intel.com/en-us/vcsource/tools/perceptual-computing-sdk/demos
