How To Help People With Disabilities With A Computer Program

(1)

Faculty of Informatics

Computer-Human Interfaces

Personal information technologies

András Lőrincz

Neural Information Processing Group

http://nipg.inf.elte.hu

(2)

Eötv ö s L o rá n d U n iv e rsi ty F a c u lt y o f In fo rma tics Zsolt Ákos Marci _Kati Melinda Dani Balázs II Balázs I Gergő Zoli I Gábor Gyula Szityú Viktor Zoli II András Company Research and academy Back from Columbia University Now, finishing

Thanks are due to my

to my group

(3)

Eötv ö s L o rá n d U n iv e rsi F a c u lt y o f In fo rma tics

1

• Main activites

2

• Personal information – example

3

• Typical & specific

4

• ―Dictionaries‖ and their integration

András Lőrincz http://nipg.inf.elte.hu

Content

(4)

Main activites



Machine learning



Computer-human interaction (TÁMOP projects)



Natural language processing (US Air Force Research Lab)



Machine vision



Map-Reduce optimization (Morgan-Stanley)



Analysis of customers’ behavior (Hungarian Telekom)



Neuroscience & psychology  learning from the structure of the brain

(5)

1

• Main activites

2

• Personal information – example

3

• Typical & specific

4

• ―Dictionaries‖ and their integration

Content

(6)

Motivation



Non-speaking, but speech understanding children with special needs



They can barely interact and communicate



Their intelligence, knowledge, and personality need interaction



augmentative and alternative communication



We develop enabling ICT tools for them



for communication and control

(7)

“Stone Age”

(8)

Motivation

(9)

Eötv ö s L o rá n d U n iv e rsi F a c u lt y o f In fo rma tics 9



Zozó and the first results



Monitor

(10)

The first three trials

(11)

The sequence of 20 trials

cursor trajectories _{target positions on screen}

Not perfect,

(12)

Development: Bencus’ story

Highly imprecise cursor control: barely controlled head swings

In three months he learned how to control

(13)

Motivation

Aibo webcam mike speaker

(14)

It is for real

ultrasonic RF-MEMS distance estimation RF-MEMS motes

Laptops and webcams

Practicing at the

Alternative and Augmentative Communication Center

Budapest

(15)

Using Dasher

and

playing Load Balancing game

(16)



Intelligent computer-human interface



Cognitive and emotional profiling



Intelligent dialogue system



Learning and development



virtual environments



edutainment



serious games

We / they need

(17)



Data collection and example based

recommendation systems



personalization



Interaction with other people



depersonalization



social networks, social games, social involvement

We / they also need

Viktor Gyenes

(18)

Social Gaming

(19)

EU Future and Emerging Technology

Exhibition April 2009, Prague:

Science Beyond Fiction

webcam emotional monitoring gesturing

playing ―anger‖

―Not sufficient artificial intelligence‖

(20)

Education

Two Classrooms

András Lőrincz http://nipg.inf.elte.hu Budapest Node

(21)

Under planning

“Nurse HomeCare”

(22)

Under planning

Elderly Apartment Houses:

“Gold Sunset”



Törökbálint



Újpalota



Zugló

(23)

In sum



We need to collect large databases about



previous events (i.e., movies)



results of tests



development of interactions



emotions and cognitive profiles subject to



privacy



in order to have a predictive recommendation system



for typical events

 annotation through machine recognition



for specific events

 novelty recognition through machine learning

(24)

1

• Motivation

2

• Personal information – example

3

• Typical & specific



Recommendations

4

• Outlook:

• ―Dictionaries‖ and their integration

Content

(25)

Dot patterns

July 08, 2010, Gatsby András Lőrincz http://nipg.inf.elte.hu

Basic forms

(26)

Dot patterns

Basic forms Study items

(27)

Dot patterns

Test items

(28)

Dot patterns

Test items

control

hippocampal subject

Test results on ―category learning‖

(29)

Dot patterns

Test results on recognition

control

hippocampal subject

Test results on ―category learning‖

control

hippocampal subject

(30)

normal child, age four years and two months

autistic child, age three and half years

Discrimination vs. generalization

(31)

Autism



Individuals with autism have difficulty abstracting subtle spatial information that is necessary



for the formation of a mean prototype,



for categorizing faces and objects.

(32)

Nine dots in autism

discrimination / visual acuity



Autistic group is less accurate



Number of prototype

formers is smaller

International Meeting for Autism Research May 7 - 9, 2009,

Gastgeb et al. Univ. Pittsburgh

Eagle-Eyed Visual Acuity Ashwin et al., Biol. Psych. 2008

(33)

1

• Main activites

2

• Personal information – example

3

• Typical & specific

4

• ―Dictionaries‖ and their integration

Content

(34)

Route to dictionary integration



Annotated databases:



Wikipedia, WordNet, OpenCyc

―ontologies‖



Map ontologies onto each other



―recommendation system‖



map typical parts



Map to other modalities



LabelMe: segmented images and textual information



segmented images and movies



model of the self: interaction—control—optimization

(35)

Route to dictionary integration

Movies

Motion understanding and control

Images

Image understanding through texts

Texts, documents

Dialogue system

(36)