AutoExperimenter: A tool for generating meaningful data

To automatically conduct experiments and collect signicant amounts of EM data, I designed and implemented the AutoExperimenter tool9_{. It accepts an experiment description and oers} all required functionality to fully start, parametrize, conduct, and tear down any experiment performable using software. For my work, I developed a simulation environment for a PR2 robot that performs household tasks in a kitchen setting, mainly setting a meal table with varying requirements.

To dene an experiment, I designed a very abstract description language that controls the AutoExperimenter instance's behavior. The denition is written in Yet Another Markup Lan- guage (YAML) format, and thus both easily readable by humans and interpretable by program code. In the following, I highlight the most important features.

Workers are the main element for any AutoExperimenter instance. They represent any pro- gram started to perform an experiment. This ranges from making instantly returning calls to external components down to processes that run throughout the full lifetime of the experiment. Their syntax is as follows:

8. Automated Experiments for Data Collection workers: ;; Includes all workers

- command: roslaunch ;; Starts worker with the executable `roslaunch'

parameters: [ltfnp_executive, ltfnp_simulated.launch] ;; Arguments

checklist: ;; Must match all for successful startup

- name: moveit ;; Internal identification name

matchmode: contains ;; How to match `template'

template: "All is well!"

message: "MoveIt! launched successfully" ;; Print this upon success

- name: reasoning matchmode: contains

template: "ltfnp_reasoning/prolog/init.pl compiled" message: "Reasoning started successfully"

quithooks: ;; If any of these match, tear down experiment

- name: failed_to_load matchmode: contains

template: "failed to load"

message: "A component failed to load"

timeout: 120 ;; Optional timeout; 0 for infinite wait

append-variance: true ;; Optionally append task variance to process

The command signies which executable to start a new process for, followed by its parameter list as command line arguments. From then on, AutoExperimenter checks for the presence of all checklist items. If all items were detected, the worker was started successfully. If one or more were not met within timeout seconds, the whole experiment is torn down. A checklist item is matched/found when any of the worker's output lines coincides with its template. The matching methods available are:

(◦) match: Full match of both strings

(◦) contains: The output line contains the template string (◦) beginswith: The output line begins with the template string

The quithook entries are the exact negative of checklist items: Whenever one or more match, the process is deemed unsuccessful and the experiment is torn down. Syntactically, they function exactly like their checklist counterpart.

Optionally, a worker can have the current task variance appended to the parameter list when the append-variance switch is present and set to true. How task variance is dened is described further below. All items in workers are started sequentially.

Cleaners are clearing up any leftover resources or intermediate changes the workers did. Syn- tactically, they are equivalent with workers:

cleaners: ;; Includes all cleaners

- command: kill_processes.sh ;; Start an external script

parameters: [] ;; No arguments given

checklist: - name: killed

matchmode: match

template: "Finished killing processes" message: "Processes killed"

Other than workers, cleaners have no notion of quithook elements, timeout, or append-variance. Like workers, cleaners are executed in sequence.

Data Collection and Experience-Based Learning

Task variance allows the AutoExperimenter to automatically re-parametrize an experiment in-between runs. To this end, dierent types of parameters can be dened:

task-variances: ;; Includes all variance parameters

attendants: ;; Parameter name: `attendants'

label: "Meal attendants" ;; A descriptive label

value-type: [multiple-choice] ;; Choose a random number of items

items: [["Mary", mary], ["Tim", tim]] allow-empty: false ;; Allow empty list?

default: [tim] ;; Default value if not sampling

object-availability-factor:

label: "Percentage of objects spawned based on how many are required" value-type: [percentage, range] ;; Return a percentage

value-range: [0, 2] default: 1

distribution: normal ;; Distribution type

mealtime:

label: "Meal time"

value-type: [choice] ;; Choose one random item

items: [["Breakfast", breakfast], ["Lunch", lunch], ["Dinner", dinner]] default: [breakfast]

Each variance parameter consists of a variable name, a descriptive label, a default value, and a set of variable type dependent elds. The following variable types are allowed:

(◦) multiple-choice: Choose a random number of items from the items eld. The switch allow-empty marks whether an empty list is valid. Every item is a pair of a descriptive label and an interpretable symbol.

(◦) choice: Choose a single element from the items eld (similar to multiple-choice). (◦) [percentage, range]: Return a oating point value from within the boundaries given in

the value-range eld. The distribution eld denes the distribution type used. (◦) [integer, range]: Similar to [percentage, range], but works with integer values rather

than oating point values.

Should parameter variation be deactivated in the AutoExperimenter instance, the respective default value will be assumed for every task variance parameter.

Meta Information describes who is responsible for the experiment denition, and what ver- sion of the experiment is described in the respective YAML le. Valid elds are:

meta-information:

author: <firstname lastname> email: <[email protected]> website: <url>

version: x.y.z

Changelog data shows who changed which element when. Every item denoted here includes an author, their email address, a date, and a changeset describing the changes:

changelog:

author: <firstname lastname> email: <[email protected]>

9. Summary date: YYYY-MM-DD

changeset:

- Moved to new YAML format for experiment description - Added meta information and task variances

Running an experiment for a xed maximum number of times follows a strict order of events: 1. Load experiment description.

2. Generate task variance parameters.

3. Run all workers in order. Should any fail, run cleaners, tear down experiment and go back to step 2 until maximum number of experiments reached; quit when exhausted.

4. If all workers started successfully (all checklists complete) mark the experiment as success- ful.

5. Tear down experiment, running all cleaners in order. 6. Go back to step 2 until exhausted.

To actually record EMs for every experiment, SemRec needs to be run as a worker before the actual experiment starts. This is achieved with this worker denition:

- command: rosrun

parameters: [semrec, semrec, "-q", "-s"] checklist:

- name: initialize matchmode: contains

template: "Signify: semrec init complete" message: "Semrec Initialized"

timeout: 15

A live example with all required steps for the experiments described in my work is available online10_.