Augmented Reality Tactile Maps for Pre-Journey Learning

Augmented Reality Tactile Maps

for Pre-Journey Learning

Ryosuke Ichikari and Takeshi Kurata

National Institute of Advanced Industrial Science

and Technology (AIST), Japan

Walking Assists for the

visually-impaired people

Assisting technologies when they are walking

• White cane

• Guide dog

• Assisting technologies with computer

– Remote Sensing by RGB(-D) cameras or LRF

– Voice navigation system based on localization

technology such as GNSS

Pre-Journey learning and re-viewing before/after

walking

Tactile Map

• Maps with tactile presentation for the visually-impaired

people

• UV offset printing

：

Printing transparent plastic on the

color maps

• PIAF(Picture in a flash)

：

Using heat sensitive papers

which can swell black-printed area (lines/images)

Problems about tactile map

• Difficulties for constructing mental image

of the contents with haptic sense

– Needs experience and training

• Limitation of the amount of contents

• Production issues

– Cost: UV offset printing is expensive

– Design issue

Proposed system

：

Utilizing AR for tactile map

• Augmented Reality; AR

– Seamless integration of real

world and virtual world

– Visual augmentation is main

stream, but AR concept

can be applied for audio sense

• AR Tactile Map

（

Proposed System

）

:

– Provides audio-visual feedbacks in response to

interactive operations

– Enables dynamic and multimodal contents

Requirements for AR tactile map

• Defined requirements by hearing to the

visually-impaired people

– Augmentation for the tactile map by audio representation (for completely blind people)

– Enlarging/enhancing display for people with week sight – Providing POR(Point of reference) information

– Helping comprehension of the tactile map with interactive training

– Intuitive interaction with the system

– Simple and inexpensive configuration with off-the-shelf products – Hands free arrangement

Augmentation for the tactile map by

audio representation

• Adding audio representation makes the

information more

reachable

• The amount of the contents by the

dynamic audio

representation

is not limited by physical tactile

map

• Dynamically Providing information according to

Enlarging/enhancing display for people with

week sight

• The people with week sight might perceive map

if the contents enlarged/enhanced.

• Image modifications in responding to the users

potential modifications: optical zoom, edge

enhancement, contrast adjustment, color

adjustment

Providing POR(Point of reference) information

• Target contents of the AR tactile map:

POI

（

Point of Interest

）

and

POR

(Point of Reference)

• POR

：

Specific point location where it is easily recognized

for confirming routes such as characteristic shape and

material of ground (steps, stairs, sloop, door), sound/noise,

and scent/odor)

• Integration with POR/POI data base

– Voice navigation system with POR/POI data base [Okuno et. al, CSUN2015]

– Utilizing as the communication tool between visually-impaired people and sighted people at “Mapping Party”

Intuitive interaction with the AR tactile map

• Interactivity is important for

learning

tactile maps

and

dynamic representation

of the contents

– Trial-and-error with interactive system is short cut for

learning tactile map

• Inputting interface by

actually touching

tactile

maps

– Most

intuitive

and the

simplest

– Perceiving and inputting at the same time

Requirements of system configuration

• Adopting off-the-shelf products for availability

perspective

– For widely promoting the system

– Consists of only simple and inexpensive products

• Enabling hands-free arrangement

– From the visually-impaired people opinions

– For touching as many as possible points of the tactile

map

System configuration

System configuration of the prototype system

• Tablet PC

⇒

For Visual-Audio feedbacks

• RGB-D camera

：

DepthSense DS325

（

ToF, Range

：

15

㎝

~100

㎝）

⇒For finger gesture recognition and robust extraction of hands area, measurement of the distance between the map and finger

• PC for recognition

• Arm for fixing tablet

• Tablet mount parts

Related Works

• Image to Sound Converter [Kuroda]

– Image is converted to sound by changing key and difference of L/R sound volume according to the pixel’s coordinate

– Transparent tactile sheet can be overlaid

– Display size and UI are limited by touch display

• Kiko-AR by cybernet systems

– Mobile AR system based on Metaio junaio with audio presentation

– Image recognition and text-to-sound

– Collaborating with a printing company to produce printed material with tactile presentation and AR augmentation. – Need to shoot with smartphone’s camera

Characteristics of AR Tactile Map

：

- Various sizes and types of tactile maps can be used

- Intuitive interface for the system are focused

Implementations

Server Parts

Recognition

Client side AR tactile app

Gesture command， Finger points, lon/lat

Dababase POR data POR/POI management parts POR/POI data Finger points tracking Gesture recognition RGB-D camera Tablet PC Camera Image Fixed on backside ID of Map Lat/Lon POR/POI data collection app. POR/POI, Lat/Lon data

Visual Feedback（AR,

Enlargement/Enhancement）

Accepting finger interaction

POR data

Recognition of Tactile Map Tracking Position Of the Tactile Map

Recognition of gesture

• Recognizing “Tapping”,

“Double Tapping”,

“Searching” gestures as

the interaction commands

• Algorithm for detecting “Tapping”

– Extracting hand area by RGB-D camera with background differencing of depth image and distance threshold

– Detecting state transition within time window （①→②→ ① ）

Distance between map and

finger ②

Variety of finger gestures

• “Tapping”

– For getting user’s focused points in the map

at the intended timing

– Equivalent to “Clicking” of mouse

• “Double Tapping”

– For another command

– Equivalent to “Double Clicking” of mouse

Converting coordinate from the map coordinate to

latitude/longitude

• Tapping points are detected by counting pixels with the

patterns and calculating center of gravity

• Estimated points are on the coordinate of RGB-D image,

they are required to be converted to latitude/longitude

– RGB-D images are rectified by perspective transform with known points of corners.

– Pairs of points between the RGB-D image coordinate and coordinates on latitude /longitude for the final conversion

Automatic identification and tracking of

tactile map

• Originally defined manually.

• ORB Feature point detector/local feature descriptor is

used for identifying tactile map by RGB image

– Comparing mean distance of feature descriptors

• Corners of the tactile map can be tracked by estimating

homography matrix between rectified image templates

and input image

– At lees 4 pairs are required for calculating homography matrix – RANSAC is used for robustly calculating homography matrix

Enlarging the map with text-to-speech reading

as the reaction of “Tapping” gesture

Mapping party

fo

People are gathered for

making map (Mapping Party)

Virtual Mapping Party with AR Tactile Map

Panoramic image and 3D-Sound

Workshop on Virtual Mapping Party

• Held at Miraikan, Tokyo, Japan

• March 18

-19

, 2016. 1h x 6 times

• Collecting POR/POI by experiencing panoramic images

and 3D sounds with HMD(Oculus, GearVR. Etc.), PC

and headphone

Watching panoramic image with 3D sound

for collecting POR/POI

Usage of the AR tactile map for virtual mapping

party

• Allowing the visually-impaired people to join

the mapping party by gesture

– Search: Confirming POR/POI on the tactile map with

sound for telling existence of POR/POI where user

touches

– Tap: Confirming POR/POI

with Text-to-Sound when

user taps the specific point

– Double Tap: Requesting

the POR/POI for the

Summery of a questionnaire for the AR tactile map

Comments about gesture interface

• It is fun to search POR/POI by actually touching.

• It took time to get used to “double-tap”, but finally I mastered. • It’s good concept, but needs more accuracy of detection.

• It lacks feedbacks to tell whether the system detected gesture or not. • It was difficult to use.

• Other techniques can be used for more robust detection of gesture.

• Touch panel with tactile presentation(sheets?) can be used for robustly detecting gesture.

• It is hard to distinguish between detection error and nothing there to be displayed.

Summery of a questionnaire for the AR tactile map

(cont.)

Comments about requesting function

• Requesting function is good because the it can reflect visually-impaired people’s demand.

• Filtering technique might be required for extracting important requests • It’s better to add function to send request by voice recognition.

• Currently only points of the request are defined, contents of the request might be needed (Just want to get POR/POI or feel scared)

Comments about the contents

• It seems helpful for the visually-impaired people to check obstacles such as trees.

Summery of a questionnaire for the AR tactile map

(cont.)

General Suggestions and Comments

• Scale size of the map is not fitted. Considering to accuracy of the gesture recognition, the tactile maps should be bigger in this case. • It seems interesting if it can realize dynamic tactile presentation in

some day.

• In addition to requesting, other opportunities for the visually-impaired people to join the events seem to be needed.

• Functions to increase the number of participants such as position based SNS seem to be required.

Conclusions

• AR tactile map: Audio-visual augmentation for tactile map

by RGB-D image recognition and text-to-speech.

• Audio-Visual feedbacks by AR for dynamic presentation

– Intuitive gesture interface by using RGB-D camera

• Future works

– Increasing accuracy of the gesture recognition – Covering individual difference of gesture

– Noise and unclear situation (sunlight, close to paper）

– Virtual Mapping Party with the AR tactile Map

• Acknowledgement

– This research is conducted as a part of the project sponsored by JST RISTEX.

Thank you!

• Ryosuke Ichikari, Ph.D.

– Postdoctoral Researcher at AIST

– Contact: [email protected]

プロトタイプでの実装の詳細

• Client

側

AR

触地図アプリ

– Android

アプリとして実装

–

カメラ映像を用いた触地図の

ID

とカメラ・触地図間の位置

関係の認識

–

ユーザの注目箇所をジェスチャ認識部と

UDP

通信

–

音声提示，視覚提示（拡大）等で情報提示

•

指ジャスチャ認識プログラム

– Windows

アプリとして実装

– DepthSenseSDK

を用いて

RGB-D

カメラ映像を処理

サイトワールド

2015

でのデモ

デモ展示等で判明した課題

•

カメラ・触地図間の位置関係がずれる

•

タップ動作の個人差の問題

関連事例

•

画像聴覚化装置

*(

画像聴覚化研究所

)

–

きっかけ：視覚障害者に美術館で絵画を体感できるように

–

音程，音が聞こえる方向により画像を聴覚化

–

触図シートを重ねることで，触覚も併用可能

AR

触地図：触地図の自由度，簡素な機器構成，

直観的インタフェースを重視，

画像聴覚化装置の音自体に位置情報を

AR

触地図とのインタラクション

• RGB-D

カメラを用いて触地図を触る動作を認識

•

前提：触地図と

RGB-D

カメラの位置関係は真上から

の撮影位置で固定

⇒背景差分が適用可能

•

距離チャネル（

D:depth

）を用いることで

…

–

触地図の色や手の肌の色に影響されない認識

–

触地図と手の接触の有無を認識