To model a particular task, the places in the PN are used to represent object states, while the transitions in the PN are used to represent motions, i.e. actions taken by the human demonstrator. The sequence of object states and motions observed dictates the sequence of places and transitions of the PN. This allows PN to be automatically created. The advantage of using PNs is that PNs are flexible enough to model various task flows, including synchronous and asynchronous, parallel and serial processing. Additionally, the created PNs can be used to generate a sequence of transitions for task imitation. For the purposes of modeling, it is assumed that transitions in the PN occur instantaneously [81], even though during the real task a transition may take some time to actually occur.
5.2.1
Petri Net Training
The automatic PN creation algorithm proceeds as follows:
1. When all objects are stationary, the state of each object is represented using a place in the PN. If the place representing a particular state does not already exist, a new place is added to the PN to represent the state of the object. To determine if a place already exists or not a simple search through all previously seen states is performed. 2. When one or more of the objects are moving, object positions are recorded in order
to obtain the trajectories of the motion.
3. When a motion has just finished, i.e. objects are no longer moving, the motion is first modeled via a DMP and then identified using AP using the motion recognition approach described in Chapter 4. If the motion that has just finished is a new, pre- viously unseen motion, it is added as a new transition to the PN. A new motion is determined by searching through all the representative exemplars of previous mo- tions. Currently, the proposed algorithm only considers the case when one object is moving at a time. Smaller motions of other objects are assumed to be due to occlusions when one object is moving.
4. Then, the state (i.e. place in the PN) right before the motion and the state right after the motion are checked for each object. If the states are different, the states will be connected via the transition representing the motion just observed, i.e. the state right before the motion is added to the input place(s) and the state right after the motion is added to the output place(s) of that transition. Additionally, the object that is
moved during the motion, i.e. the object undergoing the most displacement, will also have its state prior to the motion and its state right after the motion connected via the transition, regardless of whether these states are different. This will cover cases when the object’s position before and after the motion is the same.
5. Return to step 1 or stop if all stationary and motion sequences in the observation have been processed.
This PN creation algorithm is able to accommodate multiple demonstrations of the same task, since the same sequence of object states and motions will be observed during the task, the PN creation algorithm will be able to recognize that the places and transitions representing the object states and motions are already present in the PN and do not need to be added. In the case that there are variations in how the same task is performed during the demonstration, the created PN will either already reflect the variation or the PN will be extended to include new ways of performing the task. The PNs created for the learning system in this thesis are 1-bounded (also referred to as safe), i.e. at most one token per place at any given time, and unweighted, i.e. all the weights are effectively 1.
Lastly, separate from the PN, but as part of the task model, each object that has been observed is stored along with its affordances, i.e. the actions that have been performed on each object. This is done so that it is possible to verify whether a transition can occur given a particular object for task imitation. A transition is only possible on an object if it has previously been observed being performed on the object during demonstration.
5.2.2
Petri Net Generation
For imitating the observed task, the created PN can be used to generate a sequence of motions. This is accomplished using pre-existing PN analysis tools as follows:
1. Use an image of the initial and desired final states of the task to be imitated as input. 2. Extract the objects and object states from the input images and convert the extracted
states to a marking of the PN.
3. Using the initial marking as the root node, generate the reachability tree for the PN. 4. Using depth first tree traversal2, go through the reachability tree to find a path
2Depth first tree traversal is used because it simultaneously generates a path from the root node (initial
marking) to the tree node with the desired final marking, which indicates the sequence of transitions, i.e. actions, to be performed during task imitation.
between the root node (initial marking) and a tree node with the final marking. Always traverse the entire tree and store all potential paths found.
5. Select the shortest path found. If multiple shortest paths exist, pick the first one found. If additional criteria for task imitation are available, a more complex path selection system could be used.
The path found via the reachability tree will indicate the sequence of transitions, and thus, the motions, required to imitate the task. The DMPs will then be used to generate the actual motion trajectories, which will be used by the robot to perform the task. It should be noted that because the reachability tree of the PN covers all the places, i.e. object states, in between the initial and final states, it should be possible to perform any sub-component of the task using a PN that is created from one demonstration of the entire task.