Tutorial: Using Intel ® RealSense™
Technology in the Unreal Engine* 3 – Part 2
Setting Up Visual Studio 2010 for the Example Game
The steps below set your map file as the default map of the Example game by modifying the .ini file.
1. Go to <UE3 Source>\ExmapleGame\Config.
2. Open DefaultEngine.ini and change as below.
[URL]
MapExt=umap Map=test.umap
LocalMap=BLrealsense_Map.umap TransitionMap=BLrealsense_Map.umap EXEName=ExampleGame.exe
DebugEXEName=DEBUG-ExampleGame.exe GameName=Example Game
GameNameShort=EG
3. Open ExampleEngine.ini and change as listed.
[URL]
Protocol=unreal Name=Player Map=test.umap
LocalMap=BLrealsense_Map.umap LocalOptions=
TransitionMap=BLrealsense_Map.umap MapExt=umap
EXEName=ExampleGame.exe
DebugEXEName=DEBUG-ExampleGame.exe SaveExt=usa
Port=7777 PeerPort=7778
GameName=Example Game GameNameShort=EG
4. Open the UE3 Visual Studio project or solution file in <UE3 source>\Development\Src – UE3.sln, or open UE3.sln in Visual Studio.
Figure 1: Microsoft Visual Studio* 2010.
5. Build and run as in the previous steps. You will see the Unreal initial window and your game.
Using the Coordinate System in Unreal Engine
Before linking with the Intel RealSense SDK, it is important to understand the coordinate system in Unreal.
Position is tracked by the X-Y-Z axis (Refer to “Origin” and “RotOrigin” class in UE3 source codes) and rotation is by Euler (P-Y-R) and Quaternion (Refer to https://en.wikipedia.org/wiki/Quaternion for more detail).
Figure 2: Coordinate system.
Quaternion has one scalar factor and three vector factors.
To convert from Euler angle to Quaternion:
X-Y-Z angles:
Autoexpand Setup for a Debugger in Visual
Studio 2010 (Optional)
The debugging symbol for bone structure array, position, and rotation array was originally encrypted and unrecognizable in Visual Studio. To see debugging symbol, follow the steps below.
1. Find your Autoexp.dat
For Visual Studio and Windows 7 64-bit, it is located at C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Packages\Debugger
2. Find the debugging script and open it.
UE3 source\ Development/External/Visual Studio Debugging/AUTOEXP.DAT_addons.txt 3. Copy each [AutoExpand] and [Visualizer] section into your Autoexp.dat.
Intel® RealSense™ SDK Enabling on Unreal Engine 3
This section describes Intel RealSense SDK-related changes in Unreal Engine 3 after installing the Intel RealSense SDK and Depth Camera Manager. Face landmark and head-pose tracking of Intel RealSense SDK is used to manipulate facial expression and head movement of the example character. Head-pose tracking is intuitive since the roll, yaw, and pitch values can be used in Unreal Engine 3 as is, but face landmark tracking is more complicated.
Figure 3: Roll-Yaw-Pitch.
There are 76 traceable points in the face that the Intel RealSense SDK provides. Each expression, like blink or mouth open, has a value range with relevant points. For example, when the eye is closed, the distance between point 12 and point 16 will be around 0, and when the eye is open, the distance will be greater than 0 and variant for the personal distinction.
Based on this, the current implementation is based on the relative calculation of the minimum/maximum value between the character and the user. For example, for blinking, calculate and apply how much distance the game character’s eye should have for eyes open and closed according to the user.
Figure 4: Face landmarks and numbers of the Intel® RealSense™ SDK.
<UE3> is the home folder where UE3 is installed. Below four files are to be modified.
<UE3>\Development\Src\UnrealBuildTool\Configuration\UE3BuildConfiguration.cs
<UE3>\Development\Src\UnrealBuildTool\Configuration\UE3BuildWin32.cs
<UE3>\Development\Src\Engine\Inc\UnSkeletalMesh.h
<UE3>\Development\Src\Engine\Src\UnSkeletalComponent.cpp
UE3BuildConfiguration.cs (Optional)
public static bool bRealSense = true;
RealSense relevant codes are enclosed with “#if USE_REALSENSE” phrase. This sentence is used for defining “#if USE_REALSENSE” phrase at UE3BuildWin32.cs file. If you modify this to “false”, all of
RealSense relevant code won’t be refereed to be compiled. This is an optional.
UE3BuildWin32.cs
if (UE3BuildConfiguration.bRealSense) {
SetupRealSenseEnvironment();
}
void SetupRealSenseEnvironment() {
GlobalCPPEnvironment.Definitions.Add("USE_REALSENSE=1");
String platform = (Platform == UnrealTargetPlatform.Win64 ? "x64" :
"Win32");
GlobalCPPEnvironment.SystemIncludePaths.Add("$(RSSDK_DIR)/include");
FinalLinkEnvironment.LibraryPaths.Add("$(RSSDK_DIR)/lib/" + platform);
if (Configuration == UnrealTargetConfiguration.Debug) {
FinalLinkEnvironment.AdditionalLibraries.Add("libpxc_d.lib");
} else {
FinalLinkEnvironment.AdditionalLibraries.Add("libpxc.lib");
} }
The definition of “USE_REALSENSE” that will be used to enable or disable Intel RealSense SDK relevance at source codes (Optional).
Since Unreal Engine 3 is a makefile-based solution, the Intel RealSense SDK header file and library path should be added at the project’s include and library path.
UnSkeletalMesh.h
#if USE_REALSENSE
PXCFaceData* faceOutput;
PXCFaceConfiguration *faceConfig;
PXCSenseManager *senseManager;
void InitRealSense();
void ReleaseRealSense();
#endif
This is the declaration part of the Intel RealSense SDK classes and functions. The bone structure manipulating part is at UpdateSkelPos() of UnSkeletalComponent.cpp.
UnSkeletalComponent.cpp
#if USE_REALSENSE
#include "pxcfacedata.h"
#include "pxcfacemodule.h"
#include "pxcfaceconfiguration.h"
#include "pxcsensemanager.h"
FLOAT rsEyeMin = 6;
FLOAT rsEyeMax = 25;
FLOAT rsMouthMin = 5;
FLOAT rsMouthMax = 50;
FLOAT rsMouthWMin = 40;
FLOAT rsMouthWMax = 70;
FLOAT chMouthMin = -105;
FLOAT chMouthMax = -75;
……
#endif
Include header files of RealSense SDK. Defines minimum/maximum values of the user and game characters, starting with “rs” as the user’s value, and “ch” as the game character’s value (this should be changed according to the user and game character’s appearance). For example, for blinking, this defines how much distance a game character’s eye should have for eyes open and closed according to the user.
void USkeletalMeshComponent::Attach() {
……
#if USE_REALSENSE
senseManager = NULL;
InitRealSense();
#endif
The Attach() function calls the InitRealSense() function to initialize Intel RealSense SDK’s relevant classes and configure the camera.
#if USE_REALSENSE
void USkeletalMeshComponent::InitRealSense() { if (senseManager != NULL) return;
faceOutput = NULL;
senseManager = PXCSenseManager::CreateInstance();
if (senseManager == NULL) {
// error found }
PXCSession *session = senseManager->QuerySession();
PXCCaptureManager* captureManager = senseManager->
QueryCaptureManager();
The InitRealSense() function configures which camera will be used,and creates face-relevant class instances.
void USkeletalMeshComponent::UpdateSkelPose( FLOAT DeltaTime, UBOOL bTickFaceFX )
{
……
#if USE_REALSENSE
if (senseManager->AcquireFrame(false) >= PXC_STATUS_NO_ERROR) { faceOutput->Update();
int totalNumFaces = faceOutput->QueryNumberOfDetectedFaces();
if (totalNumFaces > 0) {
The UpdateSkelPose() function is used for head pose and face landmark tracking.
// Head
FVector v(yaw, roll, pitch);
LocalAtoms(6).SetRotation(FQuat::MakeFromEuler(v));
LocalAtoms(6).NormalizeRotation();
Head-pose tracking is intuitive because roll, yaw, and pitch values from the Intel RealSense SDK can be used as is.
Figure 5: Face landmarks and numbers that are used for eyes and mouth expression.
To express blinking, landmark points 12, 16 and 20, 24 are used, and 47, 51, 33, 39 are used for mouth expression (detail implementation depends on developers’ preference).
// Mouth
FLOAT mouthOpen = points[51].image.y - points[47].image.y;
mouth = chMouthMax - (mouthOpen - rsMouthMin) * mouthRatio;
mouthOpen = points[47].image.x - points[33].image.x;
rMouthWOpen = chMouthWMin + (mouthOpen - rsMouthWMin) * mouthWRatio;
mouthOpen = points[39].image.x - points[47].image.x;
lMouthWOpen = chMouthWMin + (mouthOpen - rsMouthWMin) * mouthWRatio;
cMouth = chMouthCMax - (mouthOpen - rsMouthWMin) * mouthCRatio;
// Left Eye
FLOAT eyeOpen = points[24].image.y - points[20].image.y;
lEyeInner = chEyeInnerMin + (eyeOpen - rsEyeMin) * innerEyeRatio;
lEyeOuter = chEyeOuterMin + (eyeOpen - rsEyeMin) * outerEyeRatio;
lEyeUpper = chEyeUpperMin + (eyeOpen - rsEyeMin) * upperEyeRatio;
lEyeLower = chEyeLowerMin + (eyeOpen - rsEyeMin) * lowerEyeRatio;
// Right Eye
eyeOpen = points[16].image.y - points[12].image.y;
rEyeInner = chEyeInnerMin + (eyeOpen - rsEyeMin) * innerEyeRatio;
rEyeOuter = chEyeOuterMin + (eyeOpen - rsEyeMin) * outerEyeRatio;
rEyeUpper = chEyeUpperMin + (eyeOpen - rsEyeMin) * upperEyeRatio;
rEyeLower = chEyeLowerMin + (eyeOpen - rsEyeMin) * lowerEyeRatio;
BN_Lips_Corner_R, BN_Lips_Corner_L, BN_Jaw_Dum is used for mouth expression, and
BN_Blink_UpAdd, BN_Blink_Lower, BN_Blink_Inner, BN_Blink_Outer is used to express eye blinking.
(Refer to the “Facial Bone Structure in Example Characters” section for each bone number.) // Mouth
FVector m(90, 0, mouth);
LocalAtoms(59).SetRotation(FQuat::MakeFromEuler(m));
LocalAtoms(57).SetTranslation(FVector(mouthWXZ[2], rMouthWOpen, mouthWXZ[3])); // Right side
LocalAtoms(58).SetTranslation(FVector(mouthWXZ[4], lMouthWOpen * -1, mouthWXZ[5])); // Left side
// Left Eye
LocalAtoms(40).SetTranslation(FVector(eyeXY[0], eyeXY[1], lEyeUpper)); //
Upper
LocalAtoms(41).SetTranslation(FVector(eyeXY[2], eyeXY[3], lEyeLower)); //
Lower
LocalAtoms(42).SetTranslation(FVector(eyeXY[4], eyeXY[5], lEyeInner)); //
Inner
LocalAtoms(43).SetTranslation(FVector(eyeXY[6], eyeXY[7], lEyeOuter)); //
Outer
// Right Eye
LocalAtoms(47).SetTranslation(FVector(eyeXY[8], eyeXY[9], rEyeLower)); //
Lower
LocalAtoms(48).SetTranslation(FVector(eyeXY[10], eyeXY[11], rEyeOuter)); //
Outer
LocalAtoms(49).SetTranslation(FVector(eyeXY[12], eyeXY[13], rEyeInner)); //
Inner
LocalAtoms(50).SetTranslation(FVector(eyeXY[14], eyeXY[15], rEyeUpper)); //
Upper
void USkeletalMeshComponent::ReleaseRealSense() { if (faceOutput)
faceOutput->Release();
faceConfig->Release();
senseManager->Close();
senseManager->Release();
}
Close and release all of the Intel RealSense SDK relevant class instances.
Facial Bone Structure in Example Characters
In the example, the face is designed with 58 bones. In the image, each box represents a bone. A complete list of bones follows.
Figure 6: Names of bones.
Bone name Bone number Bone name Bone number
BN_Head 0 BN_Tongue02 29
BN_Eye_Dum_L 1 BN_Tongue03 30
BN_Blink_UpAdd_L 2 BN_Lips_Lower_Dum 31
BN_Blink_Lower_L 3 BN_Lips_Lower_R 32
BN_Blink_Inner_L 4 BN_Lips_Lower_C 33
BN_Blink_Outer_L 5 BN_Lips_Lower_L 34
BN_Eye_L 6 BN_Nose_Dum 35
BN_Eye_Lower_L 7 BN_Noseball_Dum 36
BN_Eye_Dum_R 8 BN_Noseball_R 37
BN_Blink_UpAdd_R 9 BN_Nose_C 38
BN_Blink_Lower_R 10 BN_Nose_l 39
BN_Blink_Inner_R 11 BN_Noseball_L 40
BN_Blink_Outer_R 12 BN_Nose_R 41
BN_Eye_R 13 BN_Nosetip 42
BN_Eye_Lower_R 14 BN_Eyebrow_Center_R 43
BN_Lips_Upper_Dum 15 BN_Eyebrow_Outer_R 44
BN_Lips_Upper_R 16 BN_Eyebrow_Inner_R 45
BN_Lips_Upper_L 17 BN_Eyebrow_Outer_L 46
BN_Lips_Upper_C 18 BN_Eyebrow_Center_L 47
BN_Lips_Corner_R 19 BN_Eyebrow_Inner_L 48
BN_Lips_Corner_L 20 BN_Cheek_Lower_R 49
BN_Jaw_Dum 21 BN_Cheek_Upper_R 50
BN_JawTip 22 BN_Cheek_Lower_L 51
BN_Jaw_FL 23 BN_Cheek_Upper_L 52
BN_Jaw_BL 24 BN_Ear_R 53
BN_Jaw_FR 25 BN_Ear_L 54
BN_Jaw_BR 26 BN_Mouth_Upper_L 55
BN_Jaw 27 BN_Mouth_Upper_R 56
BN_Tongue01 28 BN_Nose_Lower 57
Conclusion
To make an avatar that moves and copies users’ facial movements and expressions to enrich the gaming experience in UE3 and using the Intel RealSense SDK, implementation of the UE3 source code is the only option, and developers must know which source file to change. We hope this document helps you when making avatar in UE3 with the Intel RealSense SDK.
About the Authors
Chunghyun Kim is an application engineer in the Intel Software and Services Group. He focuses on game and graphic optimization on Intel® architecture.
Peter Hong is an application engineer at the Intel Software and Services Group. He focuses on enabling the Intel RealSense SDK for face, hand tracking, 3D scanning, and more.
For More Information
Epic Unreal Engine
https://www.unrealengine.com Intel RealSense SDK
http://software.intel.com/realsense Notices
Intel technologies’ features and benefits depend on system configuration and may require enabled h ardware, software or service activation. Performance varies depending on system configuration. Che ck with your system manufacturer or retailer or learn more at intel.com.
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.
This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.
The products and services described may contain defects or errors known as errata which may cause deviations from published specifications. Current characterized errata are available on request.
Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm.
This sample source code is released under the Epic’s UE3 License Agreement.
Intel, the Intel logo, Intel Core, and Intel RealSense are trademarks of Intel Corporation in the U.S.
and/or other countries.
*Other names and brands may be claimed as the property of others.
© 2016 Intel Corporation.
