Developing with Oculus: Mastering the Oculus SDK. Volga Aksoy Software Engineer, Oculus

(1)

SPEAKER SPEAKER

Volga Aksoy Michael Antonov

Mastering the Oculus SDK

| |

Software Engineer, Oculus

Chief Software Architect, Oculus

(2)

CONNECT CONNECT

New SDK Features

New SDK Features:

(4)

Headset shows up as an OS Display

• Application must place a window on Rift monitor

• Icons and Windows in a wrong place

• Windows compositor handles Present

• There is usually at least one extra frame of latency

• More latency if no CPU & GPU sync is done

(5)

Outputs to Rift without Display being a part of the desktop

• Headset not seen by the OS

• Avoid jumping windows and icons

• Decouples Rift v-sync from OS compositor

• Avoids extra GPU buffering for minimum latency

• Use ovrHmd_AttachToWindow

• Window swap chain output is directed to the Rift

Hope to see Direct Mode become the longer term solution

(6)

• Direct to Rift Supports duplicating

the image to a Window

• Minimum overhead

• Can be disabled by

ovrHmdCap_NoMirrorToWindow

• Possible to display a different image

instead, such as messages in this screen

(7)

LibOVR shims Direct3D/GL Runtime

• Must call ovr_InitializeRenderingShim() before D3D use

• Window Swap Chain has an associated Rift Primary Buffer

• On DK2, back buffer is copied and rotated to Primary for display

• For best quality, use reported resolution

User/Kernel Mode Video Driver Shim

• Hides Rift from OS, allowing us to find and render to it

• Allows flipping with Rift's v-sync (no composition queue)

(8)

Symptom: “Rift doesn’t turn on”

Optimus (NVidia + Intel HD GPU Laptops)

• Discrete GPU not connected to HDMI port

• SW has to create a bridge and copy data

• Use NVidia driver bridge in Win 7/8; Win 8.1 cross adaptor support

• Works on Razer Blade laptops

• On some machines copy is slow (10ms+ making it unusable)

Multi-GPU (SLI/Crossfire)

• Rift plugged into the “wrong” adapter, need to copy

(9)

DisplayLink

• Conflicting driver shim; working with Display Link to resolve.

• Too slow for Rift output.

Multi-Monitors

• May sync to wrong Display => V-sync tear or Judder

• Resolved with upcoming driver update

Other Software Shims

• ASUS Splendid color, etc.

• Plan: Adding debugging logic, fix incrementally

(10)

Move Distortion/composition to a Separate Process

• Remove Shims from the Application

• Avoid state-related rendering issues

• Support Time warp layer composition

• Asynchronous Time Warp (TBD)

Multi-GPU TW rendering

• Implementing Time warp on secondary GPU with Optimus

Researching better GPU pipelining / preemption

(11)

New SDK Features:

Latency Testing &

(12)

VR Motion-To-Photon Latency

Input

USB

Game

Engine

Pixel

Switching

New

Image

Write

Display

(13)

• Keeping latency low is crucial for a solid VR experience

• Goal is < 20 ms, and hopefully close to 5 ms

• We’re almost there! (stay tuned for some numbers)

(14)

Latency Testing in DK1

• Required custom hardware:  

Oculus Latency Tester

• App goes into “latency testing”

mode

• Too cumbersome, ultimately not

very practical

(15)

• DK2 can:

• Sample top-right pixel color • Time stamp of color change

• Messaged tohost PC via USB

• SDK handles rest of the latency

orchestration on host PC

(16)

• SDK can internally time everything up to the point of calling Present()/

SwapBuffers() (i.e. pre-present).

• Post-present timing is a black-box to the VR app

• SDK relies on DK2 latency testing for post-present timing

(17)

Latency Testing in DK2

Variable frames primarily due to differences in:

• OS

• GPU

• Render APIs

(18)

At Oculus, we’ve seen post-present latencies From 0 to 4+ frames

That’s a delta of: 50+ ms @ 75 Hz!

(19)

High-level implementation

• PC renders top-right pixel with given color & records time

• HMD pings PC with detected color & time stamp of pixel

• SDK matches up detected color & calculates time difference

• Latency is reported after enough successive matches

(20)

Final latency calculated by SDK as:

Time difference from final HMD pose sampling to present/swap

+

Post-present delay duration from DK2 latency tester

(21)

• If (for any reason) timing fails

• SDK falls back to HMD-specific best-guess value

• Latency info accessible to VR app

• See OculusWorldDemo HUD info (‘Space’ to view)

(22)

(23)

Timing info can be “N/A” because:

• Not full-screened on HMD

• Not running v-sync'ed

• Unaccounted for gamma curve in

back-buffer (sRGB, gamma 2.2 etc.)

(24)

• “Ren:” Latency from point left-eye

HMD pose was queried from SDK

(25)

• “TWrp:” Latency from EndFrame()

(26)

• “TWrp:” Latency from EndFrame()

• “PostPresent” Latency from the

point GPU executed “Present()” / “SwapBuffer”

(27)

• Captured from Mac Book Pro w/

GeForce GT 750M on Windows 8.1

• ~0.0 ms Present, w/ TW at <7 ms

• Mid-display-scan lands @ ~10 ms

• We can now force display

*IMMEDIATELY after “Present()”! (*when using direct-mode)

(28)

• Extended-mode, “PostPresent” will

usually be multiple of 13.3 ms

• PostPresent will increase “Ren” &

“TWrp”

(29)

So we have motion-to-photon latency from SDK Then what?

Remember with 0.2 SDK, you provided a prediction time offset for every

HMD pose query?

Enter “dynamic prediction”

(30)

• 0.4 SDK internally knows how much prediction the app needs to stabilize

head tracking

• VR app no longer needs to worry about prediction.

(31)

New SDK Features:

(32)

• Artifact called “2-frame-rise-delay”

• Display fails to hit requested

luminance levels fast enough

• Accounting for this in 0.4 SDK

(33)

Pixel Luminance Overdrive Pix el driving v ol tage Pix el tr ansmission 1 frame Time Transmission after one frame period

Over Driving Pix el driving v ol tage Pix el tr ansmission 1 frame Time Transmission after one frame period

(34)

Current formula in distortion shader code:

overdriveColor = newColor + (newColor - prevColor) * overdriveScale;

• For-each-RGB, newColor > prevColor, overdriveScale = 0.1

• For-each-RGB, newColor < prevColor, overdriveScale = 0.05

• Currently in D3D10/11. Coming to OpenGL and D3D9.

(35)

• SDK needs to hold onto last “presented” frame

• Requires *v-sync 

*otherwise GPU won’t know the contents of display

• Yet another reason to make sure v-sync is always on

(36)

• Current implementation fixes many  

2-frame-rise artifacts

• e.g. in OculusWorldDemo, look through windows

outside,

• or outside at the cypress trees against the sky

• Except the staircase and fireplace

(37)

• “Black smear”

• Anomalous behavior in “2-frame-rise-delay” response

• “Extremely-dark-green” → “slightly-dark-green” transitions

Pixel Luminance Overdrive

• Evaluating fine tuned LUTs for the

“anomalous zone”

• Replace overdrive scale factors by

(38)

(39)

• SDK rendering vs. App rendering

• Troubleshooting Judder

• FOV, texture resolution

• Asymmetric/symmetric frustum issues

• Split vs. Shared Eye Textures

• Multi-threading and timing

(40)

Real-time Engine Integration Tips:

(41)

• 0.2 SDK didn’t do any rendering

• Only provided parameters needed for proper

rendering

• New rendering backend in SDK 0.4

• Takes over critical rendering features • App gives SDK L & R eye textures to

ovrHmd_EndFrame()

(42)

SDK 0.4 finishes rendering with the following features:

• Barrel distortion with chromatic aberration & time warp • Internal latency testing & dynamic prediction

• Low-latency v-sync and flip (even better with direct-to-rift) • Pixel Luminance Overdrive

• Health & Safety Warning

(43)

• Our Goal: Allow every engine to use SDK rendering

• Unity 3D, Unreal Engine 4 use SDK rendering

• All SDK rendering code is open source, and we highly encourage

experimentation.

• If your engine cannot use SDK rendering, then *please*  

let us know why not

• Some valid reasons exist…

(44)

• Don’t have your engine’s source code (e.g. XNA)?

• Need to support new render API? (e.g. AMD’s Mantle)?

• Need a stereo 3D post-fx with distortion (a la CryEngine)?

• Need to do post-distortion rendering (e.g. debug info)?

• Show stopper bugs or missing features in our SDK?

(45)

• In some cases, what you think is unsupported by our SDK, might actually

be supported, but we still want to know!

Ping us. We will listen.

(46)

Recommended SDK integration path:

• Follow OculusRoomTiny for bootstrapping the SDK

• Demonstrates both SDK rendering & App rendering 

• Then move onto OculusWorldDemo for advanced features

(47)

Real-time Engine Integration Tips:

(48)

• aka “Jitter-in-time”

• Very few complained about judder on the DK1

• DK2 has low-persistence + precise tracking & visuals

• Tracking hitches magnified when working accurately

(Think “uncanny valley” of VR tracking)

(49)

• Low fps: Too much work per-frame (provide GFX options!) • Highly variable prediction: Unpredictable frame loads

• Bad refresh rate: Not v-syncing with the HMD display • Bad multi-threading: Incorrect usage of timing in SDK

(50)

Real-time Engine Integration Tips:

(51)

• SDK provides recommended eye texture resolution

• Eye buffer texture resolution can be modified per-frame

• Pushed into ovrHmd_EndFrame()

• SDK will account for changes during distortion rendering

• e.g. Can drop eye resolution for better perf

(52)

• Runtime modification can be tested in OculusWorldDemo

• Hit ‘Tab’ & enable “Render Target->Dynamic Res Scaling”

• Will get better with layered distortion rendering for UI

• (e.g. high-res UI vs. lower-res 3D scene)

(53)

• SDK provides recommended FOVs

• FOV port: Left, Right, Top, Bottom tangent angles

• Helpers to convert FOV Port to projection matrix

• Please: Do not cook up your own projection matrix that  

“looks good enough” using your own magic constants.

• We have seen games with divergent stereo-parallax.

BAD!

FOV & Eye Texture Resolution

Symmetric Projection

(54)

• FOV can be set based on your game’s needs

• Set via ovrHmd_ConfigureRendering()

• SDK will generate new distortion mesh

• … and account for changes during distortion rendering

• Changing FOV every frame not recommended

(55)

• HMDs have locked physical FOV

• Unlike classic monitors, not an

arbitrary setting

• Virtual FOV settings larger than

HMD physical FOV are *generally wasteful

*could modify FOV to account for “time warp filling”

(56)

• Modified virtual FOV respects

physical FOV

• Otherwise, the view will look like  

it’s viewed from *binoculars 

(*should be a separate “zooming” feature)

(57)

FOV & Eye Texture Resolution

Outer FOV angle > Inner FOV angle?

• In real-life nose limits inner FOV

• Better HMD screen utilization by providing larger

total horizontal FOV with less frustum overlap

• DK1 FOVs more asymmetric than DK2

(58)

Real-time Engine Integration Tips:

(59)

• Some engines always expect a symmetric camera frustum

• Many sub-systems can be fixed, but can be cumbersome

• Camera frameworks with only symmetric FOV options • Deferred rendering interpolation artifacts

• Culling and LOD algorithms • Many more…

(60)

• Make FOV port symmetric

Inner FOV & Outer FOV =

max(Outer FOV, Inner FOV)

(61)

• Make FOV port symmetric

Inner FOV & Outer FOV =

max(Outer FOV, Inner FOV)

• Clipped-symmetric frustums similar

to asymmetric-frustums

(62)

Caveat:

• Wasted pixel draw on distortion

clipped area

• Perf issue? Scissor 3D scene render

to visible distortion area.

(63)

• Culling can be expensive

• Especially if done once for each HMD

eye camera

• Some engines cull once from

“center eye” & live with culling artifacts

(64)

• A “unionized” HMD camera will

allow culling once without artifacts

• Let us know if you’d want a helper

function for this

• Takes in: 2x camera transforms  

& 2x projections

• Returns: a camera transform  

& projection

(65)

Real-time Engine Integration Tips:

(66)

To share or not to share?

• SDK allows splitting two eye textures vs. sharing a single render target for

the eye textures

• ovrEyeRenderDesc provides viewport for each scenario

• OculusWorldDemo option “Shared RenderTarget”

(67)

Combined Eye Texture - Pros:

• Per-draw alternate-eye rendering

without switching render targets

• Pipelining on some GPU types

• Can save state & constant uploads

• Single-quad post-fx on both eyes

(68)

Split Eye Texture - Pros:

• If needed separable work load

• Smaller texture per-distortion

• Can modify eye texture resolutions

independently w/o wasting space

• Accidental cross-eye sampling with

time warp not a problem

(69)

Real-time Engine Integration Tips:

(70)

• Input devices are usually the first thing the game processes

• HMD also an input device!

• Some multi-threaded engines have 2+ frame input latency

• @ 75 Hz, that’s 25+ ms latency given!

• Please use time warp no matter what!

(71)

• e.g. “producer/consumer” two-thread engine

• i.e. main & render thread

• On main thread sample HMD with thread-safe functions:

• ovrHmd_GetFrameTiming() + ovrHmd_GetTrackingState()

(72)

• New API call coming soon: ovrHmd_GetEyePoses()

• Returns HMD Pose + L & R Eye Poses • No need for “viewAdjust” by VR app

• Can be used on main thread & re-used on render thread

(73)

• 0.4 SDK time warp only fixes up orientation latency

• To decrease positional latency:

• Resample HMD pose via ovrHmd_GetEyePose(s) before 3D-scene draw on *render

thread*

(74)

• Time warp needs precise timing triggers on render thread

• ovrHmd_BeginFrame() or ovrHmd_BeginFrameTiming() • ovrHmd_EndFrame() or ovrHmd_EndFrameTiming()

• No other API call directly affects SDK timing each frame!

(75)

• Time warp parameters sent to SDK via ovrHmd_EndFrame()

• ovrPosef renderPose[2] & ovrTexture eyeTexture[2]

• Important: Time warp doesn’t care about when and  

via what SDK call renderPose[2] was generated

• ovrHmd_GetTrackingState(), ovrHmd_GetEyePose() etc.

(76)

Looking Into The Future:

(77)

• Re-projects rendering for a later point

in time

• Done simultaneously with distortion

• Reduces perceived latency

• Accounts for DK2 rolling shutter

• 0.4 SDK only handles orientation,

positional possible

(78)

0.4 SDK supports synchronous time warp

• Each frame does Distortion + TW after both eyes render • Time warp uses later sensor reading

Judder if we don’t hit frame rate

• Dropping a frame can be very uncomfortable to the user • Much worse than low-fps on regular monitors

Can we use Time warp to fill in frames?

• Run game engine at half rendering speed…

(79)

• OculusWorldDemo has features to

show Time Warp in action.

• Hit ‘C’ to pause rendering 3D scene

• Time Warp will continue to “do its

thing”

(80)

• You can keep rotating the HMD to

look around in the last 3D scene render

• Artifact: Missing scenery is black

• Artifact: UI “warps” with the rest of

the 3D scene

(81)

Time warp in a separate thread Best hope for reducing Judder

• Fill in the frames based on old rendering

Implemented on Android for Gear VR

• Works well for smoothing head tracking on 30 fps game

Requires:

• GPU Priorities

• GPU Preemption

(82)

Async Time Warp – Sample Diagram

CPU Eye Render Thread CPU Time warp Thread

Frame N Frame N+1 Frame N+2 Frame N+3

Left Right Left R-‐1 Left Right L-‐1 Right TW R-‐2 Left TW TW L-‐2 Right TW Left Right

GPU Command Processor

TW

(83)

Current OS/Windows limitations

• No fine-grained GPU Preemption

• GPU rendering priorities are not exposed

Working with GPU Vendors

• AMD supports compute shader preemption

• NVidia supports GPU context priorities internally

Exploring “Adaptive” and “Cooperative” Techniques

• Generating extra TW data just in case • Checking TW time from game code

(84)

May need separate time warp transformations for:

• World-space

• FPS avatar/cockpit-space • Static UI-space

Allows feeding controller input into world-space time warp while avatar-world-space time warp works as usual

Solution: Layered time warp composition

(85)

Looking Into The Future:

(86)

Driver Improvements & Fixes

• Multi-Monitor fixes to v-sync related  

latency & judder

• Reduce post-present latency

• Handle Optimus and multi-GPU scenarios

Linux Support

Asynchronous/Adaptive Time warp

• Reduce Judder by generating extra frames when needed

Time warp Layers

• Handle Cockpit and Overlays separately from the scene

(87)

Runtime DLL

• Better forward compatibility with HW & Runtime Updates • Less frequent API versioning

Advanced VR Composition (TBD)

• Support Layered VR composition with time warp • Better state management & Preemption

• Overlays from secondary application (launcher etc.)

Audio

• See Brian Hook’s talk on VR Audio

(88)

Questions?

Michael Antonov | [email protected]

(89)

O C U LU S CO N N EC T TM

Developing with Oculus: Mastering the Oculus SDK. Volga Aksoy Software Engineer, Oculus

Mastering the Oculus SDK

Table of Contents

New SDK Features:

New SDK Features:

Latency Testing &

VR Motion-To-Photon Latency

Input

USB

Game

Engine

Pixel

Switching

New

Image

Write

Display

Latency Testing in DK1

Latency Testing in DK2

+

Enter “dynamic prediction”

New SDK Features:

Pixel Luminance Overdrive

Real-time Engine Integration Tips:

Real-time Engine Integration Tips:

Real-time Engine Integration Tips:

BAD!

FOV & Eye Texture Resolution

FOV & Eye Texture Resolution

Real-time Engine Integration Tips:

Real-time Engine Integration Tips:

Real-time Engine Integration Tips:

Looking Into The Future:

Async Time Warp – Sample Diagram

Looking Into The Future:

Questions?