Example AR image. Augmented Reality. Augmented Reality. Milgram s Reality- Virtuality continuum. Why Augmented Reality? Is AR easier/harder than VR?

(1)

Augmented Reality

Matt Cooper

(Many slides based on the MUM2003 Tutorials by Mark Billinghurst and Mark Ollila)

Milgram

Milgram’

’s

s

Reality-

Reality

-Virtuality continuum

Virtuality continuum

Mixed Reality

Reality - Virtuality (RV) Continuum Real Environment Augmented Reality (AR) Augmented Virtuality (AV) Virtual Environment

Adapted from Milgram, Takemura, Utsumi, Kishino. Augmented Reality: A class of displays on the reality-virtuality continuum

Augmented Reality

Virtual Reality: Replaces Reality

–

–Immersive Displays Immersive Displays

Augmented Reality: Enhances Reality

–

–SeeSee--through Displaysthrough Displays

Characteristics

–

–Combines Real and Virtual ImagesCombines Real and Virtual Images –

–Interactive in realInteractive in real--timetime

– –Registered in 3DRegistered in 3D

Example AR image

Youngkwan Cho, STAR system

Why Augmented Reality ?

Virtual Reality is Ideal for:

–

–Replacing the Real WorldReplacing the Real World –

–Simulation, Training, GamesSimulation, Training, Games

Augmented Reality is Ideal for:

–

–Enhancing the real worldEnhancing the real world –

–Sophisticated interaction in the real worldSophisticated interaction in the real world

““Intelligence AmplificationIntelligence Amplification””

Is AR easier/harder than

VR?

Rendering: easier Rendering: easier ––ThereThere’’s less of it!s less of it!

–

–But we need faster updatesBut we need faster updates

Display (resolution, FOV, Display (resolution, FOV, colourcolour): easier?): easier?

Tracking and sensing *Much* harder:Tracking and sensing *Much* harder:

–

–Greater bandwidth requirements (video, range Greater bandwidth requirements (video, range data, etc.)

data, etc.) –

–Support occlusion, general environmental Support occlusion, general environmental knowledge

knowledge –

–A big problem for registration!A big problem for registration!

Portability:Portability:

–

–VE: User stays in one place –VE: User stays in one place –in the in the VEVE –

(2)

Additional problems of

Additional problems of AR

AR

Computer graphics: faster updatesComputer graphics: faster updates

–

–Objects must appear in the right place in the Objects must appear in the right place in the real world

real world

Tracking must be:Tracking must be:

–

–more accuratemore accurate

With respect to the real worldWith respect to the real world –

–FasterFaster

Stay aligned with the real worldStay aligned with the real world

So artificial objects are correctly So artificial objects are correctly ‘‘registeredregistered’’

A Brief History of AR (1)

19601960’’s: Sutherland / s: Sutherland /

Sproull

Sproull’’ssfirst HMD first HMD

system was see

system was see--throughthrough

Early 1990Early 1990’’s: Boeing coined the term s: Boeing coined the term ““AR.AR.”” Wire Wire

harness assembly application

Early 1990Early 1990’’s: UNC ultrasound projects: UNC ultrasound project

1994: Motion stabilized display1994: Motion stabilized display

1994: Fiducial tracking in video see1994: Fiducial tracking in video see--through ARthrough AR

A Brief History of AR (3)

1996: UNC hybrid magnetic1996: UNC hybrid magnetic--vision trackervision tracker

1998: Dedicated conferences begin1998: Dedicated conferences begin

Late 90Late 90’’s: Collaboration, outdoor, interactions: Collaboration, outdoor, interaction

2000: Augmented sports broadcasts2000: Augmented sports broadcasts

Applications

Medicine

Manufacturing

Training

Architecture

Museum

Medical

“

X

-

ray vision

”

for surgeons

Aid visualization, minimally

-

invasive

operations. Training. MRI, CT data.

–

–Ultrasound project, UNC Chapel Hill.Ultrasound project, UNC Chapel Hill. Courtesy

UNC Chapel Hill

(3)

Assembly and

Maintenance

© 1993 S. Feiner, B. MacIntyre, & D. Seligmann, Columbia University © 1996 S. Feiner, B. MacIntyre, &

A. Webster, Columbia University

Applications: annotated

environment

Public and private annotations

Aid recognition,

“

extended memory

”

–

–Libraries, maps [Fitzmaurice93]Libraries, maps [Fitzmaurice93] –

–Windows [Columbia]Windows [Columbia] –

–Mechanical parts [many places]Mechanical parts [many places] –

–Reminder notes [Sony, MIT Media Lab]Reminder notes [Sony, MIT Media Lab] –

–Navigation and spatial information accessNavigation and spatial information access

Annotation pictures

M. Haupt, & E. Solomon, Columbia University

Application: broadcast

augmentation

Adding virtual content to live sports broadcastsAdding virtual content to live sports broadcasts

–

–“First down“First down””line in American footballline in American football –

–Hockey puck trails, virtual advertisementsHockey puck trails, virtual advertisements –

–National flags in swimming lanes in 2000 OlympicsNational flags in swimming lanes in 2000 Olympics

Commercial applicationCommercial application

–

–Princeton Video Image is one companyPrinceton Video Image is one company

http://www.pvihttp://www.pvi--inc.cominc.com//

Broadcast Examples

Key AR Technologies

Input

–

–Tracking technologies Tracking technologies –

–Input devicesInput devices

Output

–

–Display (visual, audio, Display (visual, audio, haptichaptic)) –

(4)

AR Displays

Optical see

Optical see-

-through HMD

through HMD

Virtual images

from monitors

Real

World

Optical

Combiners

Optical see

Optical see-

-through

through

HMDs

Sony Glasstron

Virtual Vision VCAP

Video see

Video see-

-through HMD

through HMD

Video

cameras

Monitors

Graphics

Combiner

Video

Video see

Video see-

-through HMD

through HMD

MR Laboratory’s COASTAR HMD

(Co-Optical Axis See-Through Augmented Reality) Parallax-free video see-through HMD

Strengths of optical AR

Simpler (cheaper)

Direct view of real world

–

–Full resolution, no time delay (for real world)Full resolution, no time delay (for real world)

–

–SafetySafety

–

–Lower distortion Lower distortion

(5)

Strengths of video AR

True occlusion

–

–rather than rather than compositedcompositedas in opticalas in optical

Digitized image of real world

–

–Flexibility in compositionFlexibility in composition –

–MatchableMatchabletime delaystime delays –

–More registration, calibration strategiesMore registration, calibration strategies

Wide FOV is easier to support

Head Mounted Displays

(HMD)

Display and Optics mounted on HeadDisplay and Optics mounted on Head

May or may not fully occlude real worldMay or may not fully occlude real world

Provide fullProvide full--color imagescolor images

ConsiderationsConsiderations

–

–Cumbersome to wearCumbersome to wear –

–BrightnessBrightness –

–Low power consumptionLow power consumption –

–Resolution limitedResolution limited –

–Cost is quite high?Cost is quite high?

Maintenance especiallyMaintenance especially

The Virtual Retinal

Display

Image scanned onto retinaImage scanned onto retina

Commercialized through Commercialized through MicrovisionMicrovision

–

–Nomad System Nomad System --www.mvis.comwww.mvis.com

Video Monitor AR

Video

cameras

Monitor

Graphics

Combiner

Video

(Stereo

glasses)

Brains and Bricks

Brains and Bricks…

…

AR

interface for visualizing sensor data

–

–Using portable video seeUsing portable video see--through devicethrough device

–

–Commonly available technology.Commonly available technology.

A mobile phone.

Projector

Projector-

-based AR

based AR

Examples:

Raskar, UNC Chapel Hill Inami, Tachi Lab, U. Tokyo

Projector Real objects with retroreflective covering User (possibly head-tracked)

(6)

Example of projector

Example of projector-

-based AR

based AR

Ramesh Raskar, UNC Chapel Hill

Projection screen

Projection screen AR

AR

Place static (angled?) glass screen

(window) between user and real world

Project on screen with (angled?)

displays

Align displayed objects with real world

by tracking user

’

s head

–

–Or by other means?Or by other means?

Projection Screen

Projection Screen AR

AR

Projector User (possibly head-tracked) Real objects ‘Window’ Virtual object

Head Mounted Projector

Head Mounted ProjectorHead Mounted Projector

–

–JannickJannickRolland (UCF)Rolland (UCF)

RetroRetro--reflective Materialreflective Material

–

–Potentially portablePotentially portable

AR Tracking

The importance of

tracking

Tracking is the basic enabling

technology for Augmented Reality

–

–Realistic merged realRealistic merged real--virtual environmentvirtual environment

Tracking is significantly more difficult

in AR than in Virtual Environments

–

–Greater precision is requiredGreater precision is required –

(7)

Sources of registration

errors

Static errorsStatic errors

–

–Optical distortionsOptical distortions –

–Mechanical misalignmentsMechanical misalignments –

–Tracker errorsTracker errors –

–Incorrect viewing parametersIncorrect viewing parameters

Dynamic errorsDynamic errors

–

–System delays (largest source of error)System delays (largest source of error)

For an For an ‘‘arms lengtharms length’’display:display:

–

–1 ms delay ~ 1/3 mm registration error1 ms delay ~ 1/3 mm registration error

Types of Trackers

– – MechanicalMechanical

Armature with position sensorsArmature with position sensors

–

– ElectromagneticElectromagnetic

AC or DC field emmitors/sensorsAC or DC field emmitors/sensors

CompassCompass

– – OpticalOptical

Target tracking (Target tracking (LEDsLEDs, beads), beads)

Line of sight, may require landmarks to work well. Line of sight, may require landmarks to work well.

Computer vision is computationallyComputer vision is computationally--intensiveintensive

– – AcousticAcoustic

UltrasonicUltrasonic

–

– Inertial & dead reckoningInertial & dead reckoning

Acceleration and impulse forcesAcceleration and impulse forces

SourcelessSourcelessbut driftsbut drifts

– – GPSGPS

Outdoor Augmented RealityOutdoor Augmented Reality

Accuracy not greatAccuracy not great

Line of sight, Line of sight, jammablejammable

– – HybridHybrid

Fiducial

tracking

Since we have a real worldSince we have a real world……

……and (often) a video capture of itand (often) a video capture of it

We can use the real world to track:We can use the real world to track:

–

–Use object tracking (hard) or…Use object tracking (hard) or… –

–Use fiducialUse fiducial‘‘markersmarkers’’to provide position, scale and to provide position, scale and orientation information

orientation information

Markers

Markers…

…

Can look like anything

Can be attached to anything

May not be visible in the scene:

–

–Video Video ‘‘seesee--throughthrough’’can can overpaintoverpaintthemthem

Must be easily identified

Must be distinct and clearly

orientable

Natural Feature Tracking

Goal:

–

–Overlay virtual imagery onto normal Overlay virtual imagery onto normal printed material (maps, photos, etc)

printed material (maps, photos, etc)

Method:

–

–AR registration based on matching AR registration based on matching templates generated from image texture

templates generated from image texture

Hard to do reliably and generally

Hard to do reliably and *generally*

–

(8)

ARToolKit

Enabling technologyEnabling technology

Library for visionLibrary for vision--based AR applicationsbased AR applications

–

–Open Source, multiOpen Source, multi--platformplatform

Solves two significant problems in ARSolves two significant problems in AR

–

–TrackingTracking –

–InteractionInteraction

Overlays 3D virtual objects on real markersOverlays 3D virtual objects on real markers

–

–Uses single tracking marker Uses single tracking marker –

–Determines camera pose information (6 DOF)Determines camera pose information (6 DOF)

ARToolKitARToolKitWebsiteWebsite

http:// http://www.hitl.washington.edu/artoolkitwww.hitl.washington.edu/artoolkit//

Hardware

CameraCamera – –320x240+320x240+ ComputerComputer – –Pentium 500Mhz+Pentium 500Mhz+ –

–3D graphics video card3D graphics video card

–

–Video capture cardVideo capture card

HMD (optional)HMD (optional)

–

–Video see-Video see-through or Optical seethrough or Optical see--throughthrough

–

–Binocular or MonocularBinocular or Monocular

Typical ARToolKit System

Pentium 4 2GhzPentium 4 2Ghz--$1000$1000

GeForce4 Graphics GeForce4 Graphics --$200$200

Hauppauge Hauppauge WinTVWinTVcapture card -capture card -$50$50

Marshall Board CCD Camera Marshall Board CCD Camera --$200$200

Sony Sony GlastronGlastronPLM-PLM-A35 A35 --$400$400

VGA to NTSC converter VGA to NTSC converter --$100$100

Total Cost ~ US$1950 Total Cost ~ US$1950

ARToolKit Coordinate

Frame

Tangible AR Coordinate

Frames

ARToolKit Tracking

(9)

Tracking Limitations

Computer vision based

–

–Camera pose found only when marker is visibleCamera pose found only when marker is visible –

–Shadows/lighting can affect trackingShadows/lighting can affect tracking –

–Tracking range varies with marker sizeTracking range varies with marker size –

–Tracking accuracy varies with marker angleTracking accuracy varies with marker angle –

–Tracking speed decreases with the number of Tracking speed decreases with the number of visible markers

visible markers

An

An ARToolKit

ARToolKit

Application

Basic OutlineBasic Outline

–

– Step1. Image capture & Step1. Image capture & display

display

–

– Step2. Marker detectionStep2. Marker detection

–

– Step3. Marker identificationStep3. Marker identification

–

– Step4. Getting 3D informationStep4. Getting 3D information

–

– Step5. Object InteractionsStep5. Object Interactions

–

– Step6. Display virtual objectsStep6. Display virtual objects

AR Interaction

AR interfaces as context

based information browsers

Information is registered toInformation is registered to real

real--world contextworld context –

–Hand held AR displaysHand held AR displays

VideoVideo--seesee--through (through (RekimotoRekimoto, 1997) , 1997) Magnetic trackers or computer vision

Magnetic trackers or computer vision

InteractionInteraction

–

–Manipulation of a windowManipulation of a window into information space into information space

ApplicationsApplications

–

–ContextContext--aware information displaysaware information displays

AR Interfaces as 3D data

browsers

3D virtual objects are 3D virtual objects are

registered in 3D

–

–See-See-through through HMDsHMDs, 6 DOF , 6 DOF optical, magnetic trackers optical, magnetic trackers –

–“VR in Real World“VR in Real World””

–

–3D virtual viewpoint 3D virtual viewpoint control

control

ApplicationsApplications

–

–Visualization, guidance, Visualization, guidance, training

training

3

3D AR Interfaces

D AR Interfaces

Virtual objects displayed in 3D Virtual objects displayed in 3D physical space and can be freely physical space and can be freely manipulated

manipulated

–

–SeeSee--through through HMDsHMDsand 6DOF headand 6DOF head- -tracking are required

tracking are required –

–6DOF magnetic, ultrasonic, etc. 6DOF magnetic, ultrasonic, etc.

hand trackers for input

–

–Viewpoint controlViewpoint control –

–Traditional 3D user interface Traditional 3D user interface

interaction: manipulation, selection,

adding, removing, etc.

(10)

Augmented Surfaces

Images are projected on a surfaceImages are projected on a surface

–

–back or overhead projectionback or overhead projection

Physical objects are used as Physical objects are used as

controls for virtual objects

–

–Tracked on the surfaceTracked on the surface –

–Virtual objects are registered to the Virtual objects are registered to the physical objects

physical objects –

–Physical embodiment of the user Physical embodiment of the user interface elements

interface elements

CollaborativeCollaborative

Tangible AR: Generic Interface

Semantics

Tiles semanticsTiles semantics

– –data tilesdata tiles –

–operation tilesoperation tiles

menumenu clipboardclipboard trashcantrashcan helphelp

Operation on tilesOperation on tiles

– –proximityproximity –

–spatial arrangementsspatial arrangements –

–space-space-multiplexedmultiplexed

Space

Space-

-multiplexed

multiplexed

Interface

Data authoring in Tiles

Tangible AR: Time

Tangible AR: Time-

-multiplexed interaction

multiplexed interaction

Use of natural physical object

manipulations to control virtual objects

VOMAR Demo

–

–Catalog book:Catalog book:

Turn over the pageTurn over the page

–

–Paddle operation:Paddle operation:

Push, shake, incline, hit, scoopPush, shake, incline, hit, scoop

VOMAR Interface

Lessons Learned

Face to face collaboration

–

–AR often preferred over immersive VRAR often preferred over immersive VR

–

–AR facilitates seamless/natural communicationAR facilitates seamless/natural communication

Remote Collaboration

–

–AR spatial cues can enhance communicationAR spatial cues can enhance communication

–

–AR conferencing improves video conferencingAR conferencing improves video conferencing

–

(11)

Promising Research

Directions

Natural Feature Tracking

Outdoor AR UI Design

Other Modalities

HMD Design