Effectiveness of Haptic Feedback in Open
Surgery Simulation and Training Systems
John HUa 1, Chu-Yin CHANGa, Neil TARDELLAa, James ENGLISHb, Janey PRATTb a Energid Technologies Corporation
b Massachusetts General Hospital
Abstract. This paper presents progress in the development of an untethered haptic feedback system for an open surgery simulation and training system at Energid Technologies. A key challenge for implementing open surgery simulation is an untethered haptic feedback method. In this paper, we describe our approach to an effective untethered haptic feedback system design, current results in magnetic haptic feedback system development, and effectiveness study of haptic force feedback rendering.
Keywords. Open surgery, magnetic haptic feedback, surgery simulation [1,2,3], effectiveness, training, untethered
Introduction
Haptic feedback or touch sensation is important in surgical simulation. This may be more true in open surgery than in laparoscopic since the instruments in open surgery are not tethered to ports as in laparoscopic surgery. Very often haptics is used in open surgery where visualization is not possible. The surgeon depends on haptic feedback when identifying blood vessels under other tissues, differentiating between solid masses or fluid filled structures, gauging the amount of force being applied to an organ or tissue and in blunt and sharp dissection of tissues. Since open surgery depends on haptic feedback in an untethered instrument or directly to the surgeons hand, open surgical simulation would ideally provide the same sensations to the operator.
We have established a method for tracking surgical tools accurately and generating haptic feedback according to virtual organ models and tool-tissue interaction. With our approach, a surgical tool can be tracked by means of a camera [4], magnetic sensors and optical sensors, and the haptic force feedback can be created magnetically on the surgical tool with high fidelity [5]. This system will allow trainees to interface with a virtual surgical environment in the most natural and cognitively correct manner, namely with standard, untethered surgical tools.
This paper presents the development of an untethered magnetic haptic feedback system for an open surgery simulation and training system, a cooperative effort for the Army by personnel from Energid, MGH, and the MIT Touch Lab. In the paper, we introduce our magnetic haptic feedback system design and describe an analysis of the effectiveness of haptic feedback (tethered [6] vs. untethered) in open surgery simulation. We also explore system bandwidth requirements are report our findings.
1 Corresponding Author: Principal Robotics Scientist, Energid Technologies Corporation, 124 Mount
1. Open Surgery Simulation System of Energid Technologies
Figure 1 shows the top-level framework for the surgical trainer in open spleenectomy and Burr Hole surgery. We capture information on tools using the tracker and use that information for visualization and performance assessment. There are seven modules in the system: visual tool tracking [4] (spatial processor & object tracker), tissue deformation [6] & bone dissection, visualization, haptics, audio feedback, metrics (used for real-time quality assessment through a database of “expert” values), and a database of surgery procedures. We are exploring hardware alternatives for the optical sensors, visualization devices, and the haptic feedback system [4,5].
Spatial Processor Object Tracker
Real-Time Training Quality Assessment Visualization Surgical Procedures Database Visual Motion Capture
Quality Assessment
Database
Tissue Deformation & Bone Dissection
Models Tissue Deformation & Bone Dissection
Models Haptic Feedback
Audio Feedback
Figure 1: A Framework for the Surgical Trainer of Energid Technologies.
1.Effectiveness of Haptic Feedback
Effective haptic feedback is very important in open surgical simulation. Inaccurate haptic feedback may actually be harmful in the surgical training process. In open surgical simulation, the haptic feedback system should use tools which are identical to those used in the operating room. These tools should not be tethered to an electronic tracing device which would limit the natural tool-trainee interactions. Full degree-of-freedom haptic feedback becomes impossible to achieve when using tethered haptic devices, not to mention the inconvenience of tool changes during an open surgery procedure. Considering this, Energid’s magnetic haptic system is a natural choice for open surgery simulation. Figure 2 shows a few open surgery scenarios with untethered surgery tool applications.
In order to generate effective haptic feedback, the following conditions need to be satisfied: (1) accurate tissue or bone model; (2) high resolution of tool tracking; (3) high fidelity of haptic feedback generation, and (4) using untethered haptic devices.
In the development of surgical simulation and training systems, much work has gone into tissues and bone modeling and high fidelity haptic devices, however not enough emphasis has been placed on high resolution tool tracking and it’s influence on haptic rendering. We have found some tool tracking done in laparoscopic surgical simulation but in open surgery tool tracking requires 6 degrees of freedom.
Figure 3 below shows how tool orientation affects the haptic feedback in open surgery. Two scenarios are shown in this figure: one for tool-organ poking, another for tissue cutting. The reaction force depends on organ tissue model, tool tip position relative to the organ, and cutting or poking direction. In this case if there is only 3 dimension X-Y-Z position information available for the tool without its tool orientation information it is not sufficient to create the accurate and effective haptic feedback.
(a) (b)
Figure 3: Reaction force in tool organ interaction. (a) shows the actual force on the tool as the organ is poked by a scalpel, and (b) shows the reaction force on the scalpel as the tissue is cut.
2.Untethered Haptic Feedback in Open Surgery Simulation
An untethered magnetic haptic feedback system has been designed and is under development at Energid. This system uses our video-based surgical-tool tracking algorithm to determine the position and orientation of the tool. For haptic feedback, each tool is magnetized with a strong permanent magnet. Force will be applied to this magnet using multiple electromagnets in a movable magnetic actuator. This approach is illustrated in the figure below.
Surgical Tool Mobilized Electromagnetic Stator Electromagnetic Winding Array
Figure 4: The untethered magnetic haptic feedback system. On the right is the data for magnetic force generation in 1 D magnetic haptic device test platform.
Force Feedback Loop: The force-feedback system includes two modules: mobile stage control and magnetic force generation. The desired position signal will be provided by means of a vision-based tool-tracking module in the surgery simulator, and the desired force resulting from tool-tissue interaction will be computed using the tissue models. The desired force vector is realized by adjusting the distribution of the spatial electromagnetic field and the excitation currents in the field winding array.
Sensory Measurement: Sufficient sensory information must be provided for the magnetic haptic control system. The sensory measurement should be accurate and fast. Live video cameras and magnetic sensors, such as Hall sensors are used to capture the surgical tool motion and posture variations. We use cameras to provide spatial information of tool-tissue interaction in a relatively low bandwidth, and the Hall sensors and optical sensors will be considered for high bandwidth in the local control loop of the haptic system.
Actuator Positioning Loop: The mobile stage expands the effective motion range for the magnetic haptic system. The typical motion range for an effective free space magnetic force interaction is limited. It is desirable to control the mobile stage so that the electromagnetic stator can always follow the magnetized tool and the surgery tool tip stays close to the central point of the electromagnetic field, and hence sufficient magnetic interaction force can be created.
Magnetic Force Control: The right figure of Figure 4 shows the force generation with a magnetic haptic device test platform in 1D. A maximum force 8N was generated in our experiment. Higher force can be produced by means of adjusting the structural parameters in the haptic system and the control current values.
Trainee/ Surgeon Tissue Deformation Model Thread (200Hz) Common Database (Geometry, Mechanical Properties) Haptic
Thread ThreadVisual
Haptic
Interface DisplayVisual
Graphics /Images Display Visual(30Hz) Display Force(1KHz) Hand Motion Position Information Force Rendering (1KHz) Collision Detection
Figure 5: Diagram for haptic rendering.
Haptic Rendering: Figure 5 shows a diagram for our design implementing haptic rendering. There are four modules (haptic interface, tissue deformation model, patient organ information database and visualization) and two processes (haptic rendering and a related visual rendering). In our design, we will use the following bandwidths 1000Hz, 200Hz, and 30Hz for haptic rendering, tissue deformation computation, and visualization. Our minimum requirement for the haptic rendering is chosen as 300Hz preliminarily.
3.Bandwidth Requirement for Effective Haptic Feedback
Systematic studies have not yet been reported in terms of sampling rate (bandwidth requirements) so far. It is widely accepted that a sampling rate between 300Hz-1000Hz should be used for haptic feedback [7]. We specifically set out to understand the relationship between tracking error and sampling rate. Position data was collected from a Sensible Technology PhantomTM device. The data was generated by moving the stylus in a free hand motion that mimicked what would be expected from a surgeon performing a delicate surgery. A frequency analysis of the dimension containing the highest amplitude data can be seen in upper figures of Figure 6 below. Note that 95% of the energy is within the first 1.5 Hz. Even though the Nyquist sampling theorem suggests that a sampling rate of 3 Hz will capture 95% of the information, this is not sufficient for accurate estimation of future states.
Figure 6: Tracking data bandwidth analysis. Upper left: The power spectrum of the position data. Upper right: The accumulated power, noticeable is 95% of the energy falls below 1.5 Hz. Lower left: a portion of the collected position data with the extrapolated data points shown in between the subsampled data points. Lower right: average case tracking error as a function of sampling rate, which includes both extrapolation error and
To assess the error as a function of sampling rate, 1000 Hz data was subsampled at various points and a Kalman filter was applied to the subsampled data. In between subsampled data points, the state was propagated forward assuming a constant acceleration. Figure 6 below shows a portion of the position data. The subsampled data points in the lower left figure in Figure 6 are used by the Kalman filter. In the case shown, every tenth point was taken from the 1 KHz data resulting in an effective sampling rate of 100 Hz. The extrapolated values in between the subsampled data points were computed from the last apriori state estimate returned by the filter. It is noticeable that the estimator does a fairly good job of tracking the data though there are cases of both overshoot and undershoot.
The average error was determined by comparison of the estimated measurement with the colleted data. To understand how the sampling rate impacted the error, trials were run over a span of 10Hz to 1000Hz, where 1000Hz corresponded to no subsampling of the input data. The resulting curve is shown below in lower right figure of Figure 6.
Our data analysis confirmed that the tool tracking bandwidth or minimum sampling rate of motion sensing is 300 Hz with an error bound about 0.1mm, which is consistent with our design of haptic feedback system design. For a sample rate 100Hz the tracking error bound is about 0.14mm according to our analysis result.
4.Conclusion and Future Work
In this paper, we have introduced our development progress of untethered magnetic haptic feedback system. The effectiveness of haptic feedback for open surgery simulation has been discussed, and a systematic data analysis on our experimental data of tracking has been provided. In the future, we will investigate further the effectiveness of untethered magnetic haptic system though simulation validation upon completion of the whole magnetic haptic system, and more extensive studies on the relationship of tool tracking and haptic rendering will be conducted as well.
Acknowledgements
The authors gratefully acknowledge collaboration with Dr. Mandayam Srinivasan and Dr. James Biggs at MIT Touch Lab for their support in surgery modeling and haptic feedback. The work described above has been partially supported by the U.S. Army’s Telemedicine and Advanced Technology Research Center, through the direction of Mr. Harvey Magee, Dr. Kenneth Curley, and Dr. Gerry Moses.
References
[1] A. Liu, F. Tendick, K. Cleary, and C. Kaufmann, “A Survey of Surgical Simulation: Applications, Technology, and Education,” Presence, vol. 12, issue 6, Dec. 2003.
[2] S. Dawson, M. Ottensmeyer, “The VIRGIL Trauma Training System”, TATRC’s 4th Annual Advanced
Medical Technology Review, Newport Beach, CA, Wednesday, January 14, 2004.
[3] J, Kim, S. De, M. A. Srinivasan, “Computationally Efficient Techniques for Real Time Surgical Simulations with Force Feedback,” IEEE Proc. 10th Symp. On Haptic Interfaces For Virt. Env. &
Teleop. Systems, 2002.
[4] J. English, C. Chang, N. Tardella, and J. Hu, “A vision-based surgical tool tracking approach for untethered surgery simulations and training”, MMVR 13, San Diego, IOS Press, 2005.
[5] J. Hu, (Energid Technologies), US Patent Pending: Magnetic Haptic Feedback Systems and Methods for Virtual Reality Environments, filed at June 1, 2005.
[6] H. Kim, D.W. Rattner and M.A. Srinivasan (2003). “The Role of Simulation Fidelity in Laparoscopic Surgical Training.” 6th International Medical Image Computing & Computer Assisted Intervention (MICCAI) Conference, Montreal, Canada, pp. 1-8, 2003.
[7] G. Picinbono, J. Lombardo, H. Delingette and N. Ayache, “Improving Realism of a Surgery Simulator: Linear Anisotropic Elasticity, Complex Interactions and Force Extrapolation”, Project Report, INRIA Sophia Antipolis, France, September, 2000.
