StretchSense Hand Engine User Guide

(1)

StretchSense Hand Engine User Guide

This guide will provide an overview of how to use HandEngine to connect to a hand and create an animation.

Requirements

● StretchSense Hand Engine: Already installed (see DOC-5009 in Related Documents) with IP address binding configured

● Hand Engine License Key: Email [email protected] for details ● Operating System: Windows 10 Pro

1. Setting up the Glove Wired USB/Wireless Bluetooth Connection

1. Connect your glove to the PC running Hand Engine using either a USB-MicroUSB wired connection or the StretchSense USB-Bluetooth Dongle.

a. NOTE 1_{: If connecting via the USB-MicroUSB cable, you may need to do a one time configuration of} the COM port settings in Windows Device Manager. See DOC-5002 - Setting up COM Port for Wired

USB Connection_{dongle for further instructions.}

b. NOTE 2: Windows will automatically assign a COM port number to the glove and/or USB Bluetooth dongle when they are connected to a USB port. You can find which COM Port has been assigned to a particular glove/dongle by looking at the Ports (COM & LPT) category in Windows Device Manager. When you connect the glove/dongle, the associated COM port will appear in the Ports (COM & LPT) category, and unplugging the glove/dongle will make the associated COM port disappear from the

Ports (COM & LPT) category

2. Power on the glove by pushing the button on top of the glove electronics once to enter Bluetooth mode, or twice to enter USB mode. The glove LED will blink Blue when in Bluetooth mode, and green when in USB mode.

a. When starting in Bluetooth mode, the glove LED is set to a long blue blink to signify it is searching for an available StretchSense USB-Bluetooth Dongle. If it has previously been paired with a

USB-Bluetooth Dongle and it is in range of that dongle it will automatically connect and the glove LED will switch to a short blue blink and the LED on the connected dongle will go solid blue. If this is the first time using the glove with an unpaired dongle (a dongle is unpaired if when plugged into a USB port the dongle LED is solid red), move the glove within <1 ft (<30cm) of the dongle until the red light on the dongle turns first green to indicate pairing has occurred, and then blue to indicate data is streaming from the glove.

2. Getting Started with Hand Engine

(3)

2. Once launched, Hand Engine will open in a web browser tab for you to use.

a. NOTE: StretchSense recommends running Hand Engine in a Google Chrome tab.

b. NOTE: To avoid performance degradation, StretchSense recommends you do not have any other tabs open in the web browser window that is running Hand Engine.

c. NOTE: If at any point Hand Engine exhibits persistent unexpected behaviour, you can save the state of the Hand Engine UI by clicking File→ Save Session to download a configuration JSON file, then you can reset Hand Engine by clicking the Refresh button on your browser tab which will set the Hand Engine UI back to its default state. You can then click File → Load Session and navigate to your configuration JSON to reload the previous state of the UI. This does not affect any calibration that you have done previously. If you have not restored the state of the Hand Engine UI, please note you will need to reselect the actor and calibration profile and click Train Blend/Train Pose Detection if you wish to reload a previous calibration.

3. To connect the glove to Hand Engine, expand the Glove 1 dropdown by clicking on it. a. NOTE: you can use the following process for the other Gloves to add more gloves

(4)

4. Once expanded it will display the glove controls panel for Glove 1.

5. Select USB mode to acquire live data from a glove.

6. Press Scan to populate the COM Port list and select the COM port of your USB dongle/Glove, then toggle the switch next to the COM PORT box from to On to open a connection.

(5)

7. Inspect the raw data plots in the graphical view at the bottom of StretchSense Device 1 to make sure all 15 (for MoCap Pro Original gloves)/16 (for MoCap Pro SuperSplay gloves) sensors are present and accounted for. Curling your gloved fingers will make the histograms grow taller, and vice versa.

a. NOTE: MoCap Pro Original gloves do not have a wrist sensor so you should expect this histogram to be empty.

3. Calibrating the Glove

1. To calibrate a glove first you will need to create an _{Actor and Profile which is used to save and recall} calibration profiles. Press the Add button next to the Actor field and fill in the details when prompted.

a. NOTE_{: Calibration profiles relate to a specific hand. StretchSense recommends including the hand} (i.e. “left”/”right”) in the Profile Description.

b. NOTE:_{If you have previously created an Actor and/or Profile on the same PC you are currently} using, select them from the Actor and/or Profile drop down boxes.

c. NOTE: If you have previously created an Actor and/or Profile on another PC, you can transfer this data to you current PC by:

(6)

i. Export Actor and/or Profiles from the other PC by selecting File → Export Actor or File →

Export Calibration to download a JSON file containing the Actor and/or Profile data.

1. Exporting an Actor exports all calibration profiles associated with that actor, 2. Exporting a calibration Exports that Actor but with only the current calibration

profile

ii. Copy the JSON file to a location your current PC can access.

iii. Import the Actor and/or Profile by selecting File → Import Actor or File → Import

Calibration

2. Once an Actor and Profile have been created you can start adding training poses to the calibration profile using the Pose Library Controls. You can select which poses you want to use depending on your particular application, we have provided several pose libraries to help you get started:

a. Quick: A few simple poses for good general use

b. Performance_{: A more detailed pose set for more nuanced movement} c. ASL: The complete American Sign Language set of poses

(7)

3. For this example we will use theQuick Pose set. First, ensure that Preview in the Glove controls panel is set toOff. Next, expand the Quick menu in the Pose Library Controls and select the first item on the list Paddle, ensure Auto Mode is On as this will automatically progress to the next pose on the list once the previous pose is captured.

a. NOTE: If you wish to calibrate multiple gloves to the same pose at the same time then toggle Group

Calibration to On. When Group Calibration is On, every glove that is connected to Hand Engine and Hand Engine is receiving data will be trained to the selected pose at the same time. Whether you are doing a pair of gloves on a single actor, or calibrating multiple actors simultaneously, data is captured from every glove at the same time so all of the hands/gloves/actors must be in the correct pose at the same time during capturing.

b. NOTE: Even if Auto Mode is On, you can still manually select the next pose to be trained from the Pose Library Controls using your mouse,Auto Mode will not interfere and simply jump to the next pose afterwards.

c. NOTE: If you’ve already calibrated before, make sure the Preview button is turned Off before you try to add more poses from the library to the profile.

4. With your connected glove, get the actor to match the pose in the Viewport.

IMPORTANT! For best results, have the actor initially copy the reference pose with their hand with the wrist straight. It is very important to make sure all of the joints on the actor’s hand match the reference hand pose from the pose library, if the actor cannot achieve the pose, it is better that they do not train to that pose than to train it incorrectly. Furthermore it is highly beneficial to ensure that if a particular joint is in the same position in more than one pose, that for each of these poses the corresponding joint on the actor’s hand is also in the same position. This provides higher quality data to the hand solver that results in optimal calibration performance. Once the actor is set in the hand pose, ask them to hold the hand pose steady and hold their forearm steady

while gently rotating at their wrist (e.g. like the wrist would move when using a skipping rope), and to keep rotating their wrist in this circular motion until the

capture is complete. Start the capture process by clicking the Capture button. The capture process takes <2 seconds for each captured pose.

(8)

5. Once the capture is completed, the viewport will automatically progress to the next pose (for example in the case of the Quick library the second pose is the Thumbs Up pose). Repeat Step 4 above using this new pose. Once all poses in the library have been captured the view port will stop updating, you will see the poses saved in the Captured Poses tree in the Glove 1 pane.

a. NOTE: You can delete individual poses from the Captured Poses list by right clicking on the pose and clicking Delete.

b. NOTE: You can delete all of the poses from the Captured Poses list by clicking the RESET button underneath. This empties the calibration profile but does not delete the calibration profile.

6. Next you can select how to visualise the calibrated glove:

a. Blend Mode

i. A continuous movement that interpolates between poses ii. Good for realistic hand movement

b. Pose Detection Mode

i. Snaps between the captured poses selecting the one most suitable

ii. Good for keyframing as the poses captured are the only ones that are output 7. The Quick Pose library best suits the Blend Mode, press the Train Blend Mode button to continue.

(9)

8. To view the pre-viz of your calibrated glove toggle the Preview button to On. The hand in the Hand Engine Viewport will now show the live output of the MoCap Pro glove hand capture in real time.

a. NOTE: if you like to add more poses from the pose library to your profile, you need to turn the

Preview button Off.

9. If you intend to stream hand data to your animation software, set Streaming to On. Details on how to configure your animation software to receive this data stream using StretchSense plugins are provided in separate guides that are included in your Hand Engine download.

a. NOTE: Each glove requires a unique TCP port to stream data on. By default TCP port 9000 - 9007 are used for Glove 1 - Glove 8 respectively.

10. Pose Detection Mode uses the same calibration data as Blend Mode. To enable Pose Detection Mode simply press the Train Pose Detection button. In this mode, you are able to specify a Debounce value measured in frames and transition animations using Animate Transition. This Debounce value is the number of

consecutive frames that a pose output must be stable before the pose will be applied to the hand asset. The

Animate Transition _{controls enable you to specify the type of animation (from left to right: Smooth in and}

out, linear, Explode out, Explode in) and the number of frames the animation takes.

11. Repeat the steps above for each glove you are using.

4. Configuring Timecode in Hand Engine

1. From the Master Timecode section, select the timecode source you wish to use for your glove data from the System TC dropdown box. The options available are:

a. Local System Time (default): use the system clock of the PC running Hand Engine as the timecode source. Hand Engine converts the time from the system clock of your PC into a 30fps timecode value.

(10)

b. Glove Clock: use the Glove time from each individual glove as the Hand Engine timecode source when recording data. The glove sends time measured in milliseconds since the glove connected to the PC running Hand Engine. Hand Engine converts the time to a 30fps timecode value.

i. NOTE: if you have a properly configured UltraSync One module (see DOC-5008 -

StretchSense UltraSync One Module Configuration) physically connected to your glove, the

glove will always default to output Timecode instead of time in milliseconds.

c. Glove Timecode: Use this option if you have an UltraSync One connected to each glove or have used an UltraSync One module to jam sync each glove.

d. StretchSense Circuit: Connect an UltraSync One module to a spare glove/circuit and use that single spare glove as the timecode source for all glove data. Requires selecting the COM port associated with the spare glove/circuit.

5. Setting the Local Recording Directory and Timer Based Calibration

1. Changing the directory Hand Engine will save local raw data files to: To change the root directory where files are saved:

a. First create that directory in Windows File Explorer, then copy the full path name of that directory. b. In Hand Engine, clickSettings → Preferences, then paste the path name to the Recording Location

text box.

i. NOTE: It is important to ensure a trailing backslash (“\”) character is at the end of the path name.

2. Enabling Automatic Timer Pose Capture_{: You can automatically run through the poses in a pose library} without requiring you to click Capture for each pose by using the Auto Capture Timer. Set the Auto Capture option to _{On, and specify Time between capture in seconds to hold each pose before capture begins.}

a. NOTE: StretchSense recommends starting with an 8 second delay to begin with and reducing that time as appropriate.

b. When Capture is clicked, the Capture button changes to Stop (Auto On) and:

i. A timer pops up on screen counting down from the time specified in Time between capture and the actor must adopt the displayed pose.

(11)

iii. At 0 seconds the capture process begins. The capture process takes ~1 second to complete. iv. Once the capture is complete, if Auto Mode is On, the viewport will be updated with the

next pose in the pose library and the timer will restart.

v. Steps i-iv repeats until the last pose in the pose library is captured

1. NOTE: The Auto Capture process can be stopped by clicking Stop (Auto On)

3. Browse previous recordings: Select Settings → Browse Recordings to open a Windows File Explorer window at the directory where Hand Engine will save local raw data files to.

6. Saving and Restoring the State of Hand Engine

Once you have setup each glove and setup Hand Engine in the configuration you desire, you can save the state of Hand Engine by clicking _{File → Save Session to download a JSON file of the state of Hand Engine. To restore a} previous state of Hand Engine, click File → Load Session and navigate to the previously saved JSON session file.

7. Recording MoCap Pro Glove Data using Hand Engine

This process assumes you have already connected each glove you are using to Hand Engine, created a calibration profile with the desired training poses for each glove, have selected Blend or Pose Detection mode, and have configured your Hand Engine timecode source (if applicable).

1. In order to record raw capacitance data, joint angle data and timecode data first define a Label for the recording, then toggle the Record Switch to Rec. To Stop the recording toggle the Record Switch to ‘Stop’.

(12)

a. NOTE: A subfolder is created for each take in C:\Users\<user_id>\Documents\StretchSense

HandEngine with the naming convention <take name>_<date>_<time>

b. NOTE: within this subfolder, two files are created for each glove connected to Hand Engine i. 1x Comma Separated Variable (CSV) text file with the naming convention

Raw_GLOVE-<x>_<actor name>_<profile name>_Hand_<left/right>_<take name>_<date>_<time>.csv containing (from left to right):

1. Raw capacitance data (16 columns) 2. Raw IMU data (if applicable) (10 columns) 3. Glove time (1 column)

4. Hand Engine time (1 column) 5. Solved joint angles (60 columns)

ii. 1x JSON file with the naming convention Cal_GLOVE-<x>_<actor name>_<profile

name>_Hand_<left/right>_<take name>_ <date>_<time>.json containing a snapshot of the

calibration profile used at the time the data was recorded.

8. Playing Back a Raw Data File Recorded in Hand Engine

1. Select _{File mode, then click Browse and navigate to a raw data CSV file previously recorded in Hand Engine.}

2. Select the Actor and Profile you wish to use to interpret the raw data file, and then click Train Blend or Train

Pose Detection depending on your needs.

3. Click Play to start the playback. Click Pause to pause the playback at the current frame and click Play again to resume from that point. Click the Stop button to reset the animation back to the start of the animation file.

a. NOTE: You can reprocess the raw data file using any Actor/Profile and any mode (Blend or Pose Detection). You do not need to use the same that was used for the original recording.

b. NOTE: If you wish to play several recorded raw data files simultaneously, repeat steps 1 and 2 in each of Glove 1-8 in Hand Engine as required, then click the Play button in Global File Playback. This option is useful for triggering the streaming of multiple raw data files to your animation software at the same time.

(13)

Disclaimer, Intellectual Property and Licensing

All resources including technical information, statements and recommendations are based on information StretchSense believes to be reliable, but the accuracy or completeness is not guaranteed. StretchSense is not liable for any loss of profit, revenue, savings, business, use, data (including data and design inputs), and/or goodwill or consequential, indirect, incidental or special damage or loss of any kind under or in connection with these resources.

INTELLECTUAL PROPERTY (IP)

StretchSense owns all IP related to StretchSense sensing elements and circuits (including technology and IP relating to sensing element design, sensor array design, cabling, connectors, tracks and termination points, sensing integrated circuits, sub-circuits, and circuits developed and/or supplied by StretchSense), sensing algorithms and all embedded firmware used to control the acquisition, processing and/or transmission of sensor data generated by StretchSense sensing elements and circuits, all embedded firmware and software used to sense, output, and/or control the operation of the StretchSense sensing elements and circuits, and all technology and IP relating to the manufacture and/or quality control of the forgoing, including mechanical, electronic or adhesive materials, systems and methods for attaching StretchSense sensing elements and circuits to a structure (collectively Core IP); all StretchSense garment designs (including material specifications, patterns, aesthetic and functional features, and garment assembly instructions), all technology and IP relating to physical placement and layout of StretchSense sensing elements and circuits and Core IP, and all design files or documents containing technical information relating to the design and function of a circuit (including an integrated circuit), enclosure, garment or sensing technology layout or illustration, including CAD representations, mechanical drawings, schematics, datasheets, specifications, bills of material and process instructions, including the drawings, diagrams, or CAD files that we develop StretchSense reference circuits (collectively Enabling IP); and all StretchSense proprietary software (including any software, firmware, API, SDK, for communication with StretchSense Sensing elements and/or circuits, or any software, firmware, API, or SDK for processing, interpreting, analysing data generated by StretchSense sensing elements and/or circuits, provided as a pre-compiled library or application or similar (collectively Proprietary Software), and all StretchSense reference software provided as source code, that is provided or otherwise made available by StretchSense as reference or example software, excluding any third party software); (collectively Reference Software). A limited license to Core IP integrated into items purchased or otherwise received from StretchSense is provided solely to the extent necessary to use, display, distribute, perform, offer for sale, sell, export, store, or import those same items provided that, in the case of Core IP that utilises DEAP technology licensed from Parker Hannifin Corporation (as further described below), such licence is, unless otherwise agreed in writing by StretchSense, limited to evaluation and application validation purposes, and commercial purposes in the consumer electronics, motion capture and augmented/virtual reality, and sports and fitness industries. A limited license to use, display, reproduce, distribute, perform, create derivative works of, make, have made, offer for sale, sell, export, store, import and otherwise make use of Enabling IP solely in conjunction with items purchased from StretchSense is incorporated in the purchase price of said items. License to use Proprietary Software is subject to accepting of the terms of StretchSense’s End User License Agreement (EULA) which is available on request from StretchSense. License to use, display, reproduce, distribute, perform, create derivative works of, make, have made, offer for sale, sell, export, store, import and otherwise make use of Reference Software is provided under the terms of the StretchSense Reference Software License.

LICENSING

StretchSense has licensed technology from Parker Hannifin Corporation and from Auckland UniServices Limited:

Parker Hannifin Corporation

StretchSense products that utilise certain dielectric electro-active polymer (DEAP) technology are manufactured and supplied under licence from the Quick Coupling Division of Parker Hannifin Corporation, Minneapolis, MN, USA (“Parker”). Such licence enables StretchSense to manufacture and supply Parker DEAP transducer products generally for evaluation and application validation purposes, and commercially for consumer electronics, motion capture and augmented/virtual reality, and sports and fitness industries. Commercial use in other industries is subject to Parker’s prior consent.

Auckland UniServices Limited

StretchSense products that utilise certain sensor and dielectric elastomer generator technologies are manufactured and supplied under license from Auckland UniServices Limited in Auckland, New Zealand (“UniServices”). These licensed technologies originally being created by the Biomimetics Lab at the Auckland Bioengineering Institute. Such license enables StretchSense to manufacture and supply sensor and dielectric elastomer generator products for evaluation, application validation, and commercial purposes for all fields.

STRETCHSENSE REFERENCE SOFTWARE LICENSE

Permission is hereby granted, to any person obtaining a copy of this software (“Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following condition:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

(14)

Revision Summary

Revision Date Notes

4 2020.09.21 Updated for Beta3.5

3 2020.08.21 Updated for Beta3

2 2020.07.21 Updated related docs, descriptions,record function and where to find

recorded files, note added that informs that user needs to turn of Preview off to add more poses, added disclaimer text

StretchSense Hand Engine User Guide