Situated Visualization with Augmented Reality. Augmented Reality

(1)

1

Dieter Schmalstieg Situated Visualization with Augmented Reality

Situated Visualization with Augmented Reality

Dieter Schmalstieg

Graz University of Technology, Austria

Graz University of Technology

2

Augmented Reality

• Overlay computer graphics on real world

• Example application areas – Tourist navigation

– Underground infrastructure – Maintenance – Games

Qualcomm Vuforia Simplify

[Wagner ISMAR 2008]

(2)

3

AR System Overview

Situated visualization

User input and camera

movement Pose

tracking Registration

of virtual content Virtual content

SPATIAL MODEL

Real world model

Situated Visualization Examples

Quadcopter flight planning Underground infrastructure visualization

[Zollmann TVCG 2014] [Mendez CG&A 2008]

(3)

5

What is Expected of AR today?

• Must run on standard devices – Smartphones, tablets, eyeglasses

• Localization in real-time – 30Hz, <100ms latency

• Accurate to 1cm / 1 degree

• Work anywhere – Indoors and outdoors

• Point-and-shoot

– Without any user training

6

Challenges of Outdoor AR

1. Mobile devices

– User go anywhere - big database, only mobiles – Mobiles: slow, little memory, poor GPS, tiny camera – Server Connection unpredictable, high network latency 2. Localization Database

– Wide area means large database

– Environment: feature-poor, repetitive, temporal variations – Searching the database takes long

– Creating the database takes a lot of effort

– Incremental updating of database typically unsupported

3. The user does not understand what works…

(4)

7

Approaches for AR Outdoor Localization

• How to get 30Hz pose tracking on a smartphone- class device everywhere?

• Better visual matching strategies – Sattler/Leibe/Kobbelt, RWTH

• Parallel tracking + mapping + localization – Overcome latency, scalability via server

– Collect more images to overcome narrow FOV/user

• Prune search space – Priors from sensors

– Priors from environment structure

Parallel Tracking and Mapping

Video stream

New frames

Map updates

Tracking Mapping

Tracked local pose

FAST SLOW

Simultaneous localization and mapping

(SLAM)

in small workspaces

Klein/Drummond, U. Cambridge

(5)

9

Parallel Tracking, Mapping and Localization

Video stream

New frames

Map updates

Tracking Mapping

Tracked global pose

Wide-area visual feature database

Global pose Matching New keyframes

CLIENT SERVER

FAST SLOW

SLOWEST

10

Geo-Registered Content

Video stream

New frames

Map updates

Tracking Mapping

Tracked global pose

Wide-area visual feature database

Global pose Matching New keyframes

Global pose

Geo- registered content

Annotations in local reference frame Langlotz Proc. IEEE 2014]

(6)

11

Tracking and Mapping on Mobiles

Panoramic SLAM

• Only rotation, user must stay in one place

• Works instantly

Full 6DOF SLAM

• User can move freely

• Needs baseline – walk several meters

[Wagner IEEE VR 2010]

• Localization success rate is strongly correlated to aperture angle

• Steep angles (looking down the street)  poor matching success

• Rates saturate around 180° (due to missing information at steep angles)

Panorama Localization Study

Monotonic increase of success with wider angle

[Arth ISMAR 2011]

(7)

13

Hybrid SLAM

• Both 6DOF and Panoramic SLAM

• Switch dynamically

• Triangulate later

Triangulated Infinite Features

6DOF Keyframes

Local Pano Maps 53% success

98% success

[Pirchhheim ISMAR 2013]

14

Outdoor Localization from SLAM

[Ventura TVCG 2014] [Ventura CVPR 2014]

14

(8)

15

Tracking Through Global Model Occlusion

15

Prune Search Space with Sensor Priors

• GPS: only search near position prior

• Compass: only search in approximate heading

• Accelerometer/Gravity:

Only consider features with right orientation

GPS

Gravity

Compass

[Arth ICPR 2012]

(9)

17

• 15% higher success rate of localization

• Much faster

Improvement with Sensor Priors

18

Prune Search Space with GIS

• OpenStreetMap now available everywhere!

• Offline:

– Align features with facades during reconstruction

– Reconstructed model is less distorted

• Online:

– Compute visibility sets

– Visibility from facades (GPU) – Prune database using visibility

– Detecting building outlines in image [in progress]

[Arth com.geo. 2013]

[Arth. ISMAR 2009]

(10)

19

User Generated Situated Visualization

Video stream

New frames

Map updates

Tracking Mapping

Tracked global pose

Wide-area visual feature database

Global pose Matching New keyframes

Global pose

Geo- registered content

Annotations in local reference frame

User Generated Content - Examples

Annotated environments User generated video

[Langlotz IEEE Perv. Comp. 2012] [Langlotz OzCHI 2012]

(11)

21

Future Social Augmented Reality Infrastructure

Big Content Provider

Small Personal

Servers

• Content is tagged – User name

– Content type – Position – ...

• Uploaded to a web community server

• Content can be accessed via http using tags

22

Conclusions and Future Work

• AR will establish itself as a new medium

• People have to learn how to use and understand such a new medium

• Many open challenges

– Scalable infrastructures (e.g., sustainable AR city models) – Content creation tools (tap into legacy databases like

GoogleMaps)

– Content complexity (AR use cases that provide benefit over simply QR-codes)

– Robustness of the user experience (tracking failures etc.)

– Market fragmentation (one app per AR experience)

– Ergonomics (return of the glasses?)

(12)

23

Questions?

References

• [Arth com.geo. 2013] Clemens Arth, Jonathan Ventura, and Dieter Schmalstieg. Geospatial Management and Utilization of Large-Scale Urban Visual Reconstruction. In Proc. 4th International Conference on Computing for Geospatial Research and Application (COM.Geo 2013), San Jose, CA, USA, July 2013.

• [Arth ICPR 2012] Clemens Arth, Alessandro Mulloni, and Dieter Schmalstieg. Exploiting Sensors on Mobile Phones to Improve Wide-Area Localization.

In Proc. International Conference on Pattern Recognition (ICPR) 2012, Tsukuba, Japan, November 2012.

• [Arth ISMAR 2009] Clemens Arth, Daniel Wagner, Manfred Klopschitz, Arnold Irschara, and Dieter Schmalstieg.

• [Arth ISMAR 2011] Clemens Arth, Manfred Klopschitz, Gerhard Reitmayr, and Dieter Schmalstieg. Real-Time Self-Localization from Panoramic Images on Mobile Devices. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, Switzerland, October 2011.

• [Langlotz IEEE Perv. Comp. 2012] Tobias Langlotz, Daniel Wagner, Alessandro Mulloni, and Dieter Schmalstieg. Online Creation of Panoramic Augmented Reality Annotations on Mobile Phones. IEEE Pervasive Computing, 11(2):56-63, April 2012.

• [Langlotz OzCHI 2012] Tobias Langlotz, Mathäus Zingerle, Raphael Grasset, Hannes Kaufman, Gerhard Reitmayr:

• [Langlotz Proc. IEEE 2014] Tobias Langlotz, Thanh Nguyen, Dieter Schmalstieg, and Raphael Grasset. Next Generation Augmented Reality Browsers:

Rich, Seamless, and Adaptive. Proceedings of the IEEE, 102(2):155-169, February 2014

• [Mendez CG&A 2008] Erick Mendez, Gerhard Schall, Sven Havemann, Sebastian Junghanns, Dieter Fellner, and Dieter Schmalstieg. Generating Semantic 3D Models of Underground Infrastructure. IEEE Computer Graphics and Applications, 28(3):48-57, May 2008.

• [Pirchhheim ISMAR 2013] Christian Pirchheim, Dieter Schmalstieg, and Gerhard Reitmayr. Handling Pure Camera Rotation in Keyframe-Based SLAM.

In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2013, Adelaide, Australia, October 2013.

• [Ventura CVPR 2014] Jonathan Ventura, Clemens Arth, Gerhard Reitmayr, and Dieter Schmalstieg. A Minimal Solution to the Generalized Pose-and- Scale Problem. In Proc. IEEE Computer Vision and Pattern Recognition 2014, Columbus, OH, USA, June 2014.

• [Ventura TVCG 2014] Jonathan Ventura, Clemens Arth, Gerhard Reitmayr, and Dieter Schmalstieg. Global Localization from Monocular SLAM on a Mobile Phone. IEEE Transactions on Visualization and Computer Graphics, 20(4):531-539, March 2014.

• [Wagner IEEE VR 2010] Daniel Wagner, Alessandro Mulloni, Tobias Langlotz, and Dieter Schmalstieg. Real-Time Panoramic Mapping and Tracking on Mobile Phones. In Proc. IEEE Virtual Reality, pages 211-218, Boston, USA, March 2010.

• [Wagner ISMAR 2008] Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, Tom Drummond, and Dieter Schmalstieg. Pose Tracking from Natural Features on Mobile Phones. In Proc. 7th IEEE International Symposium on Mixed and Augmented Reality, pages 125-134, Cambridge, UK, September 2008. Best Paper Award. [Zollmann TVCG 2014]

• AR Record & Replay:Situated Compositing of Video Content in Mobile Augmented Reality. Proceedings of the 24th Australian Computer-Human Interaction Conference (OzCHI) Pages 318-326, 2012.

• Wide Area Localization on Mobile Phones. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 73-82, 2009.