Programming by demonstration

ENSMA

L I S I

Programming by demonstration

A relevant paradigm for applications

that manages huge data

Laboratoire d'Informatique Scientifique et Industrielle

École Nationale Supérieure de Mécanique et d'Aérotechnique

Programming by demonstration

• Why ?

– Programming by demonstration (PbD) is not data visualization

– PbD deals with user enhancement

– PbD is a good way for giving flexibility

• Who ?

Contents

• 1- Definition

• 2- Overview

• 3- Concepts

Definition

• "If a user knows how to perform a task on a

computer, that should be sufficient to create

a program to perfom the task"

– A. Cypher, 1993

• The system spies the user to make a

program

Definition

• "My idea of how programming should work

was that it should be like teaching someone

how to perform a task"

– H. Lieberman, 2001

• The user teaches the system how to make a

program

Definitions

• Visual Programming

• Programming by/with example

• Example-based programming

• End-user programming

• Programming by demonstration

• Demonstrational intefaces

Definitions

• "End-User Programming is a form of

programming done by the end-user to customize the information processing"

– B. Nardi, 1993

• "Programming by example is a powerful end-user programming paradigm enabling computer users without formal training in programming to create sophisticated programs"

Définitions

• Demonstrational interfaces allow the user to perform actions on concrete example objects, while constructing an abstract program

– Myers, 1993

• Just in time programming is the implementing of algorithms during task-time, the time when the user is actually trying to accomplish the task

Overview of PbD

• 27 years of history

Overview: Pygmalion (1975)

Overview: Pygmalion (1975)

!

Overview: Pygmalion (1975)

!

6

?

false

true

Overview: Pygmalion (1975)

!

6

?

false

true

=

Overview: Pygmalion (1975)

!

6

?

false

true

=

Overview: Pygmalion (1975)

!

6

?

false

true

=

6

Overview: Pygmalion (1975)

!

6

?

false

true

=

6

1

Overview: Pygmalion (1975)

!

6

?

false

true

=

6

false

1

Overview: Pygmalion (1975)

!

6

?

false

true

=

6

false

1

Overview: Pygmalion (1975)

!

6

?

false

true

=

6

false

1

false

Overview: Pygmalion (1975)

!

6

?

false

true

=

6

false

1

false

Overview

1- The programming age

• 1975 "Pygmalion"

• 1976 "Autoprogrammer"

• 1978 "Exemplary programming" et

"Programming by abstract demonstration"

• 1979 "Tinker" and "Programming by

example"

Overview: SmallStar (1986)

National Parks Close Name Bryce Zion Canyonlands Proposed Size 1 Page 1 Page 1 Page 0 Dcty Version of 11/12/84 22:40:02 11/12/8422:40:10 11/12/8422:40:19 11/12/8422:40:34 Sample Close

This is some text in a text trame Form field

Button

Park States Zion Utah Vosemite California

Great Smory Mountains North Carolina Tennesse

Halbert Report Letters Calc Program Printer Directory In Out

Overview: SmallStar (1986)

National Parks Close Name Bryce Zion Canyonlands Proposed Size 1 Page 1 Page 1 Page 0 Dcty Version of 11/12/84 22:40:02 11/12/8422:40:10 11/12/8422:40:19 11/12/8422:40:34 Sample Close

This is some text in a text trame Form field

Button

Park States Zion Utah Vosemite California

Great Smory Mountains North Carolina Tennesse

Halbert Report Letters Calc Program Printer Directory In Out Run Program

Close Start Record'g Stop Record'g 1st Step Step Run

Program

Move bryce to National Parks

Move cedar Breaks to National Monuments. Copy Itinerary to the Desktop (copy is ).

Overview

2- The "Direct Manipulation" age

• 1983 "Direct manipulation"

• 1984 "Rehearsal world" et "Smallstar"

• 1986 "MIKE" and "Peridot"

• 1988 "Graphical search and replace"

• 1991 "Macros by example"

Overview: Eager

• Suppose you want to make a numbered list

of your emails (title+subject) in your text

editor

Overview: Eager

• Suppose you want to make a numbered list

of your emails (title+subject) in your text

editor

Overview: Eager

• Suppose you want to make a numbered list

of your emails (title+subject) in your text

editor

– Write '1' in the text editor

Overview: Eager

• Suppose you want to make a numbered list

of your emails (title+subject) in your text

editor

– Write '1' in the text editor

– Copy the first mail subject in the mailer – Paste in the text editor

Overview: Eager

• Suppose you want to make a numbered list

of your emails (title+subject) in your text

editor

– Write '1' in the text editor

– Copy the first mail subject in the mailer – Paste in the text editor

Overview: Eager

Overview: Eager

– Paste in the text editor

Overview: Eager

– Paste in the text editor

– Write '2' in the text editor

Overview: Eager

– Paste in the text editor

– Write '2' in the text editor

– Copy the second mail subject in the mailer – …

Overview: Eager

– Paste in the text editor

– Write '2' in the text editor

– Copy the second mail subject in the mailer – …

Overview: Eager

– Paste in the text editor

– Write '2' in the text editor

– Copy the second mail subject in the mailer – …

– Write '3' in the text editor

I guess I can help you. Do you want me to try ?

Overview: Eager

– Paste in the text editor

– Write '2' in the text editor

– Copy the second mail subject in the mailer – …

– Write '3' in the text

Overview: Eager

– Paste in the text editor

– Write '2' in the text editor

– Copy the second mail subject in the mailer – …

– Write '3' in the text

– Copy the third mail subject in the mailer

Overview

3- The Inference age

• 1988 - "Tels"

• 1990 - "Eager" et "Turvy"

• 1991 - "Mondrian"

Overview

4- The rules editing age

• "KidSim", "Cocoa" puis "Stagecast Creator"

• "AgentSheets"

• "Pavlov" vs. "Gamut"

Overview: Grammex

• Suppose you want to define a grammar to

parse an email address. Ex.

– [email protected]

• The system

lieber @ media . mit . edu

Overview: Grammex

media . mit

• New grammar

– HOST

• Select "media.mit" • The system gives its

interpretation • Correct it

• The system knows "HOST" as...

Overview: Grammex

media . mit

• New grammar

– HOST

• Select "media.mit" • The system gives its

interpretation • Correct it

• The system knows "HOST" as...

Overview: Grammex

media . mit

• New grammar

– HOST

• Select "media.mit" • The system gives its

interpretation • Correct it

• The system knows "HOST" as...

lieber @

. edu

Overview: Grammex

media . mit

• New grammar

– HOST

• Select "media.mit" • The system gives its

interpretation • Correct it

• The system knows "HOST" as...

lieber @

. edu

Overview

5- The intelligent agent age

• "Internet Scapbook" et "SmallBrowse"

• "Dynamic macros", "SmartEdit",

"Grammex", "SWYN"

Overview

6- Industrial PbD

• 1990 "LIKE" et "EBP"

• 1992 Les systèmes paramétriques

– Systèmes variationnels

– Systèmes paramétriques purs

General idea

• "A software agent records the interactions between the user and a conventional "direct-manipulation" interface, and writes a program that corresponds to the user's actions. The agent can then generalize the program so that it can work in other situations similar to, but not exactly the same as, the

examples on which it is taught."

Problem's heart

• "The main challenge confronting Programming by Demonstration is how to infer the user's intent"

– A. Cypher, 1993

• "Generalization is the central problem of PBE and, ultimately, should enable PBE to completely

replace conventional programming"

PbD Mechanisms

• Total inference

• Interactive inference

• Macro-recording

• Agents

• Modeling and PbD

Mechanisms

Total inference

• "infering from traces"

• The user gives the inputs and the expected

outpurs

• The system deduces the algorithm that is

able to realize the task

Mechanisms

Total inference

• The system is intelligent !

• Strongly limitated

– LISP programs

– Variational approach in CAD

Mechanisms

Total inference

Une famille = Un modèle + des contraintes D L 2*L /3

+

=

Avantages • Simple d'utilisation • Théoriquement illimité Inconvénients • Lenteur relative • Surtout 2D ou 2D et demi • Ne peut que varier

Mechanisms

Total inference

• Advantages

– Not "How", only "What" – Natural specifications

• Drawbacks

– Completude problems – Magic !

Mechanisms

Interactive inference

• In Example-Base Programming

– Give 2 different series of data to allow the system to infer an alternative structure

• In Peridot

– Generalizes alignment rules from approximate sketches

Mechanisms

Interactive inference

Commands Circle Edit-Picture MOUSEDependent Rectangle Run-Procedure StringFromSelect Add-Return-Stmt ChangeDrawFnc ChangeFont Conditional Exit FixActive Icon Iteration MOUSE-Exception ReCalcConstraint RefreshAll Reinitialize ResetMOUSE Selection String Write-Procedure P E R I D O T Parameters to MyMenu

Items = ("Replace" "Move" "Copy" "Delete" "Delete All "Exit" SomeItems = ("Move" "Delete"

OtherShades = (1 65 3345 34401 43405 31710 -1

CurrenItem "Move"

ActiveColors = (65 34401 43405 31710 -1 ScrollControl (2000 75

**Mouse** (-39 -12 nil nil nil **Clock** (NIL 0 active values Drawing Area V Hello there

Creating function My Menu 0

(QUOTE (PLUS 3 (Field BlackRect0369 LEFT))) (QUOTE (PLUS 3 (Field BlackRect0369 BOTTOM))) (QUOTE (PLUS -6 (Field BlackRect0369 WIDTH))) (QUOTE (PLUS -6 (Field BlackRect0369 HEIGHT))) (QUOTE (INPUT . REPLACE)))

(RUNICON.CREATE (QUOTE Icon0371) (QUOTE CheckMarkBM)

(QUOTE (PLUS (QUOTIENT (DIFFERENCE (Field WhiteRect0370

WIDTH) (Field ME WIDTH)) 2)

(Field WhiteRect0370 LEFT))

(QUOTE (PLUS (QUOTIENT (DIFFERENCE (Field WhiteRect0370

HEIGHT) (Field ME HEIGHT)) 2)

(Field WhiteRect0370 BOTTOM))) (QUOTE (INPUT . REPLACE))) (RunFillCircle.CREATE (QUOTE GrayCircle0372) 42405

(QUOTE (PLUS (PLUS 35 (Field WhiteRect0370 WIDTH)) (Field WhiteRect0370 LEFT)))

(QUOTE (PLUS (QUOTIENT (DIFFERENCE (Field WhiteRect0370

HEIGHT)

(Field WhiteRect0370 BOTTOM))) 18

User interaction

The Gray circle (GrayCircle0372) (marked with an N) seems to be centered vertically and at the right of the white rectangle WhiteRect0370] (marked with WRT) offset horizontally by 35

Mechanisms

Interactive inference

• Is it really inference?

• Do we really unknow what to do?

– We give explicitly two cases…

– We make approximate alignment…

Mechanisms

Macro-recording

• Many generalization techniques

– On objects

• Constants, variables, "iterative variables" • Naming...

– Portant sur les structures de contrôles

• Alternative • Iteration

Mechanisms

Macro-recording

• Many generalization techniques

– Explicit techniques

• A posteriori (coded editing)

• A priori (purely interactive, with menu, button)

– Implicit techniques

• With predefined rules

Mechanisms

Macro-recording

• Program representation

– Why?

• Is Macro recording complete? • Debugging?

– Textual vs. Graphic vs. No representation – Editable program vs. Non editable

Mechanisms

Agents -

1. Limited agents

• The spy principle...

• Eager

– As soon as a sequence of user actions looks repetitive, it appears…

– and proposes to help the user in ending the task – A very intrusive system...

Mechanisms

Agents -

1. Limited agents

• Dynamic Macro

– All operations are recorded in a log file (cf. Marco recording)

– On explicit request (REPEAT), the system

searches into the log file a repeated sequence, and generalizes it

Mechanisms

Agents -

2. Instructible agents

• ScrapBook, Wrapper, Grammex…

Mechanisms

Agents -

3. Programmable agents

• StageCast Creator, Cocoa, Gamut…

• A simple programming metaphor:

event-response (before/after)

• Complex behaviors are possible by

composition

Mechanisms

Towards modelisation

• One example that actually works:

parametric systems

Mechanisms

Towards modelisation

Une famille = Un modèle contraint

= D L E 2*L/3 U U U 2*L/3 Avantages • Simple d'utilisation • 3D (CSG) Inconvénients • Difficilement modifiable • Ne peut pas tout traiter : - contraintes multiples - manque de structures de

Application domains

• Programming learning

• Simulation

• Utilities

– Macros, GUI-Builder, tasks automating, instructible agents

• End-User Programming

PbD, a relevant approach

an example: biologists

• Experts

• Strong needs for flexibility

– Experimentation

– Kleenex programming

• They don't want "magic programs"

• They don't like programming (syntax, bugs)

• They are able to understand concepts

PbD, a relevant approach

an example: biologists

• PbD avoids syntactic problems

• Many non intrusive solutions exist

• Many tasks are first exploratory tasks, and

then repetitive tasks

• But, inference is not usable (magic)

Some false ideas

• PbD is inference, e.g. A.I.

• PbD is not complete programming

• PbD is restricted to some limited domains

(visual)

• PbD is only for novices

• PbD is not actually in use

Bibliography

• "Whatch What I Do: Programming by

Demonstration"

– A. Cypher, 1993, MIT Press

• "A Small Matter of Programming"

– B. Nardi, 1993, MIT Press

• "Your Wish Is My Command:

Programming by Example"

PbD, a relevant approach

• "Users are now at the mercy of software providers who delover shrink-wrapped, one-size-fits-all,

unmodifiable "applications". With PBE, users could create personalized solutions to one-of-a-kind problems, modifying and creating existing new ones, without going through the arcane

voodoo that characterizes conventional programming"