Innovate Engineering Outreach: A Special
Application of the Xbox 360 Kinect Sensor
Tanner Bryce Blair
School of Electrical and Computer Engineering The University of Oklahoma
Norman, United States [email protected]
Chad Eric Davis
School of Electrical and Computer Engineering The University of Oklahoma
Norman, OK [email protected] Abstract—In November of 2010, Microsoft released the
Kinect sensor for the Xbox 360 video game console. This device—similar to a webcam—allows an individual to interact with an Xbox 360 or a computer in three-dimensional space using an infrared depth-finding camera and a standard RGB camera. As of January of 2012, over 24 million units have been sold. Using a combination of custom and open-source software, we were able to develop a means for students to visualize and interact with the data allowing us to introduce the concepts and skills used in the field of Electrical and Computer Engineering. The unique technological application, visual appeal of the output, and the widespread ubiquity of the device make this an ideal platform for raising interest in the field of Electrical and Computer Engineering among high school students.
In order to understand the appeal of the Kinect, a working knowledge of the technical details of the device is useful. The novelty and appeal of the Kinect sensor lies in its infrared camera, which is comprised of two distinct devices. An infrared projector sends out a 640x480 grid of infrared beams, and an infrared detector is used to measure how long the reflection of each beam takes to return to the sensor. This data set is known as a “point cloud”. This point cloud is a three-dimensional vector comprised of data points between 40 and 2000, which correspond to distance from the device of each beam. The data in this array can then be parsed to construct a 3d image. The Kinect’s infrared camera operates at 30Hz, or 30 samples per second, so the device is able to deliver a frame rate that is sufficient to create the illusion of motion. This allows for the development of applications that give the user a sense of interacting in real time with the image on the screen. The unique visual appeal, novelty of interaction, and relatively easy-to-understand theory of operation make the Kinect an attractive platform for recruitment and outreach.
Using the Kinect, a recruiter is able to quickly and effectively demonstrate a range of concepts involving hardware, software, and the design process on a platform that students are familiar with and find appealing. In a short window of time they are able to show examples and
explain the fundamental principles of the system while providing tangible, meaningful, and enjoyable interactivity with the device itself. This level of approachability and familiarity is rare among highly-technical fields, and provides an excellent catalyst to develop interest in Electrical and Computer Engineering education.
Keywords—Kinect; engineering; outreach; education; Processing; natural interaction; embedded systems; Xbox 360
I. INTRODUCTION
As referenced in a prior paper from our department [1], the University of Oklahoma’s Electrical and Computer Engineering department (OU-ECE) suffered a sharp decline in undergraduate enrollment from 2004 to 2008, during which enrollment numbers dropped from 387 in 2004 to 246 in 2008. In order to rectify this, action was taken to form a student-driven recruiting effort that emphasized hands-on, demonstrative outreach communicated in a peer dynamic [2]. This recruiting and outreach program was immensely successful, as evidenced by the undergraduate enrollment rising to 399 students in 2012. During this time a number of unique innovations in OU-ECE outreach were produced. While many successful hands-on demonstrations were born out of this program, a key omission was identified in the explanation and demonstration of embedded real-time systems and their application. In order to rectify this, several demonstrations were developed around the Microsoft Xbox 360 Kinect sensor. This paper aims to provide an overview of the Kinect platform, discuss the technical details of the Kinect and our application, analyze the perceived benefits of our approach, discuss the results seen thus far, and introduce some possible ways to expand the platform to increase its effectiveness.
II. DEVICE OVERVIEW
In November of 2010, Microsoft released the Kinect sensor for the Xbox 360 video game console [3]. This device— similar to a webcam—allows an individual to interact with an Xbox 360 or a computer in 3-dimensional space using an infrared depth-finding camera and a standard RGB camera.
This device has become immensely popular, and as of January of 2012, over 24 million units have been sold [4]. There are currently over 150 games available for the Xbox that utilize the Kinect sensor, and Microsoft is currently focusing on integrating it in to the Windows environment. While Microsoft has popularized the device, the actual technology was developed by an Israel-based company named PrimeSense. PrimeSense offers a range of 3d imaging solutions; all characterized by their system-on-a-chip (SoC) design and the open-source software solutions that accompany the device.
III. HARDWARE OVERVIEW
From the perspective of an end-user the Kinect is similar to a standard webcam, but the technology behind the device is distinct from other commercially available cameras. The device is, in fact, two distinct devices—an infrared camera and a standard RGB camera—that are processed into a single dataset by PrimeSense’s SoC. The layout of the device is shown in Fig. 1.
The Kinect’s 3-dimensional imaging is accomplished by the infrared camera, which itself is comprised of two distinct parts that are managed by the SoC. The Kinect contains an infrared projector that emits a 640x480 grid of infrared beams. This grid leaves the Kinect in a conical projection and is reflected off of the environment. The Kinect then uses an infrared detector to calculate the length of time it takes each beam to return to the device. The SoC then uses that measurement to extrapolate the distance from the device at each point, and returns a three-dimensional vector to the host machine that is comprised of data points between 40 and 2000. These data points correlate directly to the number of millimeters away the person or object is from the device. This concept is illustrated In Fig. 2. In addition to creating this vector, the SoC also collects and correlates data from the RGB camera, allowing the device to determine with reasonable accuracy what color is present at each point in the cloud.
Another key aspect of the Kinect’s functionality is that it operates at 30 Hz, or 30 samples per second. This allows the device to deliver a frame rate that is sufficient to create the illusion of motion, and allows for the development of applications that give the user a sense of interacting in real time with the image on the screen.
While no single component of the Kinect is necessarily revolutionary, the combination of the components in to a complete system allows for an embedded system capable of real-time data acquisition, and enables powerful applications that are relatively simple to develop.
IV. SOFTWARE OVERVIEW
By supplying the software to access the data from the Kinect in an open source fashion, PrimeSense has successfully fostered a large developer community around the Kinect. The PrimeSense Software Development Kit consists of two essential software solutions that facilitate development with the Kinect. The first is Open Natural Interaction (OpenNI), which provides an open-source API framework for development. Specifically, the OpenNI APIs allow access to voice command recognition, hand gestures, and body motion tracking [5]. The second software solution is Natural Interface Technology for End-User (NiTE). This software acts as a simple way to use the OpenNI framework with minimal CPU load [6]. NiTE provides the tools to quickly and easily turn the data from the Kinect device in to usable information about the depth, color, IR and audio data received from the hardware [6].
When choosing the development environment there were several key features that were important to us, both as developers and recruiters. In order for it to be an effective tool for distilling the core concepts, a simple interface and a robust development environment were vital. To this end, the demonstrations were developed in the open-source development environment “Processing” [7]. This environment vastly facilitates simple, rapid development, and contains many features that are a natural fit for the Kinect. By design, “Processing” is natively capable of generating robust, visually appealing displays using intuitive commands, and has a clean, simple user interface that lends itself to being incorporated in Infrared Detector and Emitter
RGB Camera
While all beams leave the Kinect simultaneously, the beam closest to the person or object of interest returns first. The Kinect’s SOC uses the slight difference in time to create the mesh data that the software can then parse into a 3-dimensional image.
Figure 1 - Physical Layout of the Kinect
to the demonstrations. Another key compone of development environment was the Simple for “Processing” that allows for even simpl core features of OpenNI. An overview of t hierarchy is provided in Figure 3.
V. USE IN OUTREACH
Now that an understanding of the unde Kinect hardware and software has been device’s application in our outreach efforts i aim to accomplish this by providing an overv of this project, the product of our develop employed in our outreach efforts, and what p this project has yielded thus far.
In our experience, demonstrations that allo of physical interactivity serve as the best sta discussion with prospective students. While that embody this concept are abundant f embedded systems and sensing demonstrati been too complicated, for the length of tim understanding, or too simplistic to fully importance of this field of study. With these g set out to develop a series of applications and that use the Kinect to illustrate some of concepts of ECE, and the result was a collect interactive demonstrations that illustrate a topics.
To begin, we use a series of physics developed by Amnon Owed [8] that involve th able to interact with geometric shapes that fal the screen. The Kinect is capable of det outlining them, and then using that outline t manipulate the shapes on the screen using t Using edge detection, the software is able to with the ability to hold their arms out and g
Depth Physical Acquisi PrimeSense System-on-a
OpenNI NiTE Algorithms Si
Processing Development Env V User Interface
Figure 3-Software Hierarchy Overview
ent of our choice OpenNI wrapper ler access to the the development
rpinnings of the established, the is discussed. We view of the goals pment, how it is erceived benefits ow for some sort arting point for a e demonstrations for many fields, ions have either me and level of y illustrate the goals in mind, we d demonstrations the fundamental tion of engaging, wide range of s demonstrations he students being ll from the top of tecting the user,
to allow them to their movements. provide the user ather the shapes,
sweep the shapes away from them and otherwise use their physical p virtual environment. Fig. 4 featu demonstration.
Figure 4 - Physics with the Kinect
Another powerful use of the K controller for other software. T developed an application that al Ableton Live, a music creation Using the Kinect and the midi p application is able to send comma allow the user to select different v the volume, pitch, and various physical movements. This is accom detection to outline the user, and th relative positions of their limbs. T being able to move their right hand Kinect to control the intensity of th control the pitch, while using thei intensity of the reverberation or signal. A screen shot of the skeleta uses is shown in Fig. 5.
Figure 5 – Skeletal data from the Kine
ition a-chip
impleOpenNI vironment Visual Output
, throw the shapes around, presence to manipulate the ures a screen shot of this
Kinect is employing it as a To demonstrate this, we
lows the user to control and synthesis application. protocol, the “Processing”
ands to Ableton Live that virtual instruments, control sound effects using their mplished by using skeletal hen assigning values to the This translates to the user d towards or away from the he note and up or down to ir left hand to control the distortion present in the al data that this application
Both of these applications provide the pro with a novel and unique interaction, and discussion of how these demonstrations ar Typically the student is interested predomina hardware and signal chain necessary to demonstration or the software employed to interactivity, and both demonstrations lend discussion on either topic.
When the student is predominately in hardware, a quick synopsis of how the Kinect is easily delivered by explaining the infrared c of individual data points. Using “Processing show a view of the raw data and a simplisti three dimensional image that allows the stud the infrared mesh that the Kinect g demonstration in itself is typically impressiv students are surprised at the amount of inform is able to deliver. An example of the Kinects three dimensional images is shown in the scre The camera is sitting in front of the user, bu able to render the image from above.
Figure 6 - Kinect data view
During the past year, we have used these d a variety of recruiting events, both on and of and with various age levels. At a variety of tr events where our recruiters are given a table free to choose whom to talk to, the demo effective at catching the interest of prosp These demonstrations were employed at se events targeting high school students interes OU, and were very well received. Through o and interactions with students, we have com much of this success is owed to the familiari As previously noted, there are over 24 millio in the homes of consumers. This is a device th associate with having fun and playing video g this connection is enough to help convey electrical and computer engineers have on activities in their daily lives.
spective students often lead to a re accomplished. ately in either the implement the o accomplish the themselves to a nterested in the device functions camera as a mesh ” we are able to ic rendering of a dents to visualize generates. This ve, and often the mation the Kinect ability to render eenshot in Fig. 6. ut the software is demonstrations at ff of our campus traditional-format and students are onstrations were pective students. everal recruiting sted in attending our conversations me to believe that ity of the device. on Kinect sensors hat these students games, and often the impact that a wide range of
Additionally, the Kinect dem opportunity to interact with stude video games that would tradition Science as a viable major. The as brings often allow us a chance to about the hardware and lower-leve game development, and potentially could be a better fit for them. Wh interesting demonstrations that we e Kinect creates a unique starting po high school students about the deg helps attract students that would tr other areas of study.
The Kinect has proven to be an as a booth demonstration, but anoth work with the Kinect has been out to high school groups that are inter In the past year, we have used the K various school groups, including a club, a high school robotics club, a and a pre-engineering club. W mathematics club, we were ab demonstrations to the mathematical uses to determine the depth of an uses to align the color data with th other applications of mathematics robotics club we shifted our focu object detection, and the creation o With the physics club we focus models and simulations, and using t and observe how the physical wo specific set of conditions. With ea served as a powerful tool for linki current interests with a related app engineering.
Using the demonstrations outl topic of studying ECE has proven t these demonstrations over the past y immensely positive due to the d Kinect, the novelty of being able dimensions, and the prospective stu device. While no specific, quantitat for analysis at this point, our organ the methodology measure the impa begin collecting data in the coming has allowed our recruiting team to students than other demonstration allowed.
VI. FUTURE DEVELOPMENT Moving forward, there are sev would like to implement in to our heighten the effect of the demonst appeal for students interested in phy the ability to adjust simulation s
monstrations provide the ents that are interested in nally only view Computer ssociations that the Kinect speak with these students el software that goes in to y convince them that ECE hile there are a variety of employ in our outreach, the oint for conversations with gree and scope of ECE and aditionally be interested in n effective tool for outreach her important aspect of our treach targeted specifically rested in the STEM fields. Kinect demonstrations with a high school mathematics a high school physics club, When presenting to the ble to relate the Kinect l algorithms that the Kinect
object, the trigonometry it he point cloud, and various s in technology. With the
s towards machine vision, f responsive programming. sed on creating computer
those simulations to predict rld would behave under a ach subject, the Kinect has ing a prospective student’s plication in technology and ined above to broach the to be effective. As we used
year, the response has been iverse applications of the
to render objects in three udent’s familiarity with the tive data has been collected nization intends to develop act of this device and will g months. Using the Kinect reach a wider audience of ns and recruiting methods
veral key features that we Kinect demonstrations. To rations and to increase the ysics, we would like to add ettings in real time. This
would give the user the ability to select the effect of various conditions such as gravity or the mass of an object. Another possible direction that we are exploring is using the LabVIEW development environment to create functional blocks that the students could use to write their own programs within a relatively short amount of time. Using this, a recruiter could allow the student to not only experience the demonstration, but actively engage in its creation. The LabVIEW development environment also lends itself to the creation of interfaces like the ones previously described that would be used to control simulation variables. Given these advantages, the possibility of simultaneous development could be beneficial.
We would also like to work on developing the Kinect in to a series of robotics demonstrations where the actions of the user would influence the behavior of a physical object. For example, the user’s skeletal data could be used to determine the position of a servo motor, remote commands could be given through motion, or a robotic arm could be programmed to wave back when a user waves. The possibilities for effective, simple demonstrations are numerous, and we look forward to developing an increasing number of applications as our outreach efforts continue to grow.
VII. CONCLUSION
While OU-ECE’s recruiting and outreach program has seen great success over the last five years, the Kinect has enabled our recruitment team to reach a greater number of students interested in a wide array of disciplines. Using a device which most students are familiar with and have had a positive association with has allowed us to introduce the fundamentals of embedded systems to a new generation of prospective students and enabled meaningful conversation pertaining to the importance of the fields of ECE. The student excitement generated by this device at recruiting events has surpassed all other demonstrations OU-ECE has deployed in the past and it has always been the most engaging demonstration of any event it has been shown. We believe others will find the Kinect an extremely useful tool for ECE outreach.
ACKNOWLEDGMENT
As noted, many of these demonstrations were either the direct product or derivative of other members of the open source community. To that end, we wish to thank the independent developer Amnon Owed whose work was used in our physics demonstrations, Professor Max Rheiner of the Interactive Design program at Zurich University of the Arts whom contributed countless examples that aided in our development, Professor Peter Froslie of the School of Art and Art History at the University of Oklahoma for his assistance with “Processing” and communication over the midi protocol, and independent developer Jordan Kuehn who assisted with various technical challenges as they arose. All of these people were an invaluable resource, occasionally without their knowledge or against their will.
REFERENCES
[1] C. Davis, M. Yeary, and J. Sluss, "Reversing the trend of engineering enrollment declines with innovative outreach, recruiting, and retention programs," IEEE Transactions on Education, vol 55, issue 2, pp. 157-163, May 2012.
[2] C. Davis, D. Vreeland, C. Griffin, and M. Yeary, “Creating a Culture of Student-Driven ECE Recruiting and Retention”, ASEE Annual Conference and Exposition, Paper accepted and to appear in conference proceedings. June 2012.
[3] “Microsoft Xbox 360 Kinect Launches November 4” 2010. Gizmodo.
http://gizmodo.com/5563148/microsoft-xbox-360-kinect-launches-november-4. Accessed: 20 February 2013.
[4] “MICROSOFT SAYS XBOX 360 SALES HAVE SURPASSED 76 MILLION UNITS, KINECT SALES TOP 24 MILLION.” 2013. BGR.
http://bgr.com/2013/02/12/microsoft-xbox-360-sales-2013-325481/. Accessed: 20 February 2013
[5] “ABOUT OPENNI.” 2010. OpenNI.
http://www.openni.org/about/#.UWJcbJPvv78. Accessed: 24 February 2013.
[6] “NITE 2.” 2012. OpenNI.
http://www.openni.org/files/nite/#.UWJcc5Pvv78. Accessed: 24 February 2013.
[7] “Processing.” 2013. Processing.org. http://www.processing.org/
Accessed: 8 April 2013.
[8] “Amnon P5 - Experiments with Processing” 2013. .
http://amnonp5.wordpress.com/ Accessed: 8 April 2013. [9] Borenstein, Greg. 2012. Making Things See: 3D vision with Kinect,
