Models for simulating instrument-tissue interactions

(1)

Dr. U. Kühnapfel 23.01.01

Models for simulating

instrument-tissue interactions

Uwe G. Kühnapfel,

H.K. Çakmak, H. Maaß, S. Waldhausen

MMVR 2001

Newport Beach, January 27, 2001

Models for simulating

instrument-tissue interactions

Uwe G. Kühnapfel,

H.K. Çakmak, H. Maaß, S. Waldhausen

MMVR 2001

(2)

Overview

Introduction

About M.I.S. Simulation

Karlsruhe VEST System Overview

Modelling of Soft Tissue

Simulation of Surgical Interactions

Visual Effects for enhanced Realism

(3)

Motivation

Patients benefits

Patients benefits Surgeons disatvantagesSurgeons disatvantages +

+ less traumatic less traumatic - limited view- limited view +

+ fast recoveryfast recovery -- limited mobilitylimited mobility + short hospital stay

+ short hospital stay - limited haptics- limited haptics

Ü

Ü Training is essential_{Training is essential}

„Classic“

Training Systems

Pelvi-Trainer

Training with plastic ModelsTraining with plastic Models and Animal Tissueand Animal Tissue Training with living animals / cadavers

Training with living animals / cadavers

Difficult and expensive to setup, ethic problems

Ü

Ü Setup is time- and cost consumingSetup is time- and cost consuming

Ü

(4)

Bronchoscopy

Bro

Bro--NielsenNielsen et et al al. (1999), HT-. (1999), HT-MedicalMedical, USA, USA Cardiac-Surgery /

Cardiac-Surgery / AnastomosisAnastomosis Playter et al. (1997), BDI, USA

Playter et al. (1997), BDI, USA

Craniofacial Surgery Craniofacial Surgery Keeve (1996), Uni-Erlangen, Keeve (1996), Uni-Erlangen, DD Eye-Surgery Eye-Surgery

Sinclair et al. (1998), Georgia-Tech, USA

Gynaecology

Szekely et al. (1998), ETH-Zürich, CH

Szekely et al. (1998), ETH-Zürich, CH Kühnapfel et al. (1998), FZK, D

Laparoscopy

Cover et

Cover et al al. (1993), . (1993), Georgia-Tech, USAGeorgia-Tech, USA

Kühnapfel et al. (1995), FZK, D

Trauma Surgery (mil.)

Basdogan et al. (1997), Musculographics,

Basdogan et al. (1997), Musculographics,USAUSA Tumor Diagnosis and Palpation

Tumor Diagnosis and Palpation

Burdea et al. (1998), CAIP, USA

VEST-System Requirements

Soft-Tissue Soft-Tissue Simulation Simulation Training Training Environment Environment,, User interface User interface Physiology Physiology Simulation Simulation Realistic Realistic Simulation Simulation Scenarios Scenarios Interactions Interactions and Visual and Visual Effects Effects Ü

Ü Limited surgical InteractionsLimited surgical Interactions

Ü

Ü Limited anatomical Limited anatomical „„RealismRealism““

Ü

Ü Limited modelling of _{Limited modelling of}„„PhysiologyPhysiology““

(5)

Dr. U. Kühnapfel _{VEST system functional diagram} 23.01.01 Position Sensoring System Position Position Data Data Forces Forces Graphics Graphics Workstation Workstation Instrument Positions Instrument Positions Binary Switches Binary Switches SGI-IRIX /

(6)

„Phantom Box“ with user interface:

2 (3) MIS-instruments, 1 Endoscope,

6 footswitches

PC based position measuring system

Rendering and Simulation with

Rendering and Simulation with KISMETKISMET software on: software on: • SGI-workstations (Onyx, Octane) and

• SGI-workstations (Onyx, Octane) and

• NT-workstation (Intergraph, SGI-Visual PC)

Force-Feedback with commercial haptic devices:

• Impulse Engine (Immersion Corp.)

• PHANToM (SensAble Technologies)

Structure of the M.I.S. Training System Structure of the M.I.S. Training System

The “Karlsruhe Endoscopic Surgery Trainer”

(7)

(8)

Dr. U. Kühnapfel

_{VR Force-Feedback Devices}

23.01.01

Integration of two commercially available

Force-Feedback Input Devices with KISMET

• „Laparoscopic Impulse-Engine“ (Immersion)

• „Phantom“ (Sensable Technologies)

Research on „Feeling of Tissue Elasticities“

in MRI-Volume Datasets

(9)

Cakmak / Kühnapfel 23.01.01 Grasping Cutting Cutting Clipping Clipping Simulation of Soft Tissue Elastodynamics Simulation Simulation of of Soft

Soft TissueTissue

Elastodynamics

Visual Effects

Visual EffectsVisual Effects Fluids Jet of water Fluid accumulation Wave effects Air bubbles Fluids Jet of water Fluid accumulation Wave effects Air bubbles Bleeding Particle systems Texture animation Bleeding Particle systems Texture animation Stopping of bleeding Coagulation Clipping Stopping of bleeding Coagulation Clipping Smoke/Steam

Steaming up of endoscopic lense

Smoke/Steam

Steaming up of endoscopic lense

Spline-Modelling Splinecurves/2D-Cross-sections Spline-surfaces Splinecurves/2D-Cross-sections Spline-surfaces Volume Rendering

Image slice rendering 2D/3D Textures

Volume Rendering

Image slice rendering 2D/3D Textures Procedural Textures Turbulence functions L-Trees Procedural Textures Turbulence functions L-Trees Properties of tissue

Tissue measurement in vivo/vitro

Properties of tissue

Tissue measurement in vivo/vitro

Physiology Physiology Physiology Hemodynamics Simulation of Pulse Arterial bleeding Organ bleeding Hemodynamics Simulation of Pulse Arterial bleeding Organ bleeding Morphodynamics

Actively Deforming Objects Peristalsis

Dilatation Contraction

Morphodynamics

Actively Deforming Objects Peristalsis Dilatation Contraction Irrigation Suction Suction Suturing Suturing Coagulation

(10)

Elastodynamic tissue model

1. Modelling-Approach

1. Modelling-Approach: : Particle SimulationParticle Simulation

• Nonlinear, visco-elastic, mass-spring model

• System of coupled differential equations

•

• Numerical solution of ODEs Numerical solution of ODEs

)

(

)

,

(

2 2

t

f

x

t

g

dt

x

d

dt

x

d

m

i i i i i i i

⋅

+

γ

⋅

+

=

m

m_i_i Mass of knotMass of knot γ

γ_i_i DampingDamping g

g_i_i Inner forcesInner forces f

f_i_i External forcesExternal forces

(

)

i i i i t i t i i t i i t t i

m

t

B

k

g

f

m

t

v

k

v

γ

⋅

∆

+

=













−

⋅

∆

+

⋅

=

∆ +

1

Modelling of soft tissue

Lagrange Equation (ODE 2nd Order) : Newton-Euler Integration

(11)

Dr. U. Kühnapfel _{Modelling of soft tissue}_{Modelling of soft tissue} 23.01.01

Tissue-Deformation:

2.

2. Elastostatic solution of ODE-systemElastostatic solution of ODE-system

Fast Finite Element Modeling (

Fast Finite Element Modeling (FFEMFFEM))













=

























− − − −

u

f

K

i o i o 1 i 1 io 1 oi 1 o 3. 3. DeformableDeformable Volume-ModelsVolume-Models pre-calculated stiffness-matrix pre-calculated stiffness-matrix::

- very fast solver possible (++)

- recalculation required for topology

changes (e.g. cutting)

- size of stiffness matrix (--)

- import of patient data (++)

- elasticity Data from tables

- size of stiffness matrix (--)

(12)

Maaß, Çakmak, Kühnapfel 23.01.01

Mechanical Properties of soft tissue

Experiments

in-vivo and post-mortem

uniaxial tensile tests

compression tests compression tests Tissue model Tissue model Polynomial 4th order Polynomial 4th order Results Results

Different properties in-vivo and p.m.

Organ specific stress / strain curves

Linear slope for compression: 0% - 15%

Ø PhD-thesis Dr.-Ing. Maaß H., FZK - IAI, ´99

)

1 (

₂ _z ₃ _z2 ₄ _z3 z z

a

E

ε

σ

=

⋅

+

(13)

Dr.-Ing. Heiko Maaß / Kü _{Modelling of soft tissue: Elasticity Parameters}_{Modelling of soft tissue: Elasticity Parameters} 23.01.01

Speed-of-Sound and E-Modulus Range Tables

E-Modulus in MPa

0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8

Speed of Sound in m/s

Fat (soft) Fat (not soft)

Liver Spleen Heart-Muscle Kidney Fat Liver Spleen Kidney 1400 1420 1440 1460 1480 1500 1520 1540 1560 1580 1600 post mortem intra vitam

(14)

H. Çakmak / Kü 23.01.01

Modelling-Tool: KisMo (

KIS

MET-

Mo

deller)

• Interactive design of model geometry and

spatial nodal net

• Scene editor for model connectivity

• Spline modelling

• Multilayer functionality for volumetric

cell definition

• Volume rendering of CT/MR-data

(15)

H. Çakmak / Kü _{Modelling of simulation scenarios}_{Modelling of simulation scenarios} 23.01.01

KISMET

Elastodynamics Elastodynamics simulation simulation Simulation Simulation scenario scenario Elasticity Elasticity measurement measurement Medical image Medical image data data CT,MR,VF,VM CT,MR,VF,VM Internal Internal Format Format Elasticity Elasticity parameters parameters Modelling Modelling aid aid Spline-Spline-modellingmodelling ProceduraleProcedurale TexturesTextures Volume

Volume RenderingRendering

Moveable image slices

KisMo

3D-3D-GeometryGeometry

3D-3D-MassMass-Spring--Spring-NetNet

Object connections

Actively deforming

Actively deforming Objects Objects

Model

Model-/Material-/-/Material-/ and

and

Texture-Database

(16)

H. Çakmak / Kü _{Modelling of soft tissue with}_{Modelling of soft tissue with}_KisMo_KisMo 23.01.01

Modelling of deformable

objects

Methods

:

Spline

curves

and revolving

and

revolving

spline

surfaces

Object connection

:

Nearest neighbour approach

s_yp C_i s_xp s_yn s_xn y x z z_c T T P P_i_i d d_i_i n n_c_c x_c yc

(17)

H. Çakmak / Kü _{Scene Example: Laparoscopic Gynaecology}_{Scene Example: Laparoscopic Gynaecology} 23.01.01

Frame

rates

• SGI Octane, 2xR10000, 250MHz

• SGI Octane, 2xR10000, 250MHz Ü

Ü 11 11 fpsfps

•

• SGISGI Visual Visual PC PC, 2xPIII, 400MHz, 2xPIII, 400MHz

Ü

Ü 9 ₉fpsfps •

• Intergraph PCIntergraph PC, 2xPII, 400MHz, 2xPII, 400MHz

Ü

Ü 6 ₆fpsfps

Eladyn.Models Objects Mass-knots Springs Uterus,Tubes,Lig. 7 1.335 5.030 Arterial tree 6 594 2.508 Venous tree 7 675 2.844 Intestines 1 147 560 Total 21 2.751 10.942 Vertex/Polygons 15.182 6.966

Eladyn.Models Objects Mass-knots Springs

Uterus,Tubes,Lig. 7 1.335 5.030 7 1.335 5.030 Arterial tree 6 6 594594 2.508 2.508 Venous tree 7 675 2.8447 675 2.844 Intestines 1 147 560 1 147 560 Total 21 2.751 10.942 21 2.751 10.942 Vertex/Polygons 15.182 6.966

Model

(18)

Dr. U. Kühnapfel _{Modelling: Texture, Motility (Morphodynamics)}_{Modelling: Texture, Motility (Morphodynamics)} 23.01.01

(

)

_           ⋅ + ⋅ ⋅ + ⋅ ⋅ ⋅ ⋅ = fr frD rand n k wD wS t A k t _i _i i i i i i i i i,ado( , ) u sin 2π F

(

rand

)

h

A_i = 1⋅∆p_i_,_mean + sgn_rand⋅∆p_i_,_var ⋅

∑

= = 1 0 , ( , ) ) , ( i ado i ado t k F t k F •

• MathematicalMathematical FunctionsFunctions 44periodicalperiodical 44position-dependent_{position-dependent} 44time-dependenttime-dependent F_0,ado F_0,ado

Videoclip

(19)

Kühnapfel / Çakmak 23.01.01

Basic Surgical Interactions

Grasping Grasping Application of clips Application of clips Coagulation Coagulation Cutting Cutting

Surgical Interactions with deformable objects

(20)

(21)

Kühnapfel / Çakmak _{Basic interactions: application of particle systems}_{Basic interactions: application of particle systems} 23.01.01

Mass

-Spring-System

Set

Set of

of

mass

mass-

-points

points

K

and

linkage elements

V

:

FMS={K,V}

Mass

-

point

Point in

3

d

_{with final mass and infinite small volume}

Linkage elements

:

Elastic

Elastic (Spring)

(Spring)

Elastic

Bending (Spring)

Bending

(Spring)

Plastic

Viscous

(Damping

(

Damping)

)

u u u F F₀ = − ₁ =k ⋅( − l₀) pp₀₀ pp₁₁ sonst l₀ 1 0 ≥    − = − = F u F κ κ p p₁₁ p p₀₀ u F F₀ = − ₁ =η⋅& p_p₀₀ pp₁₁ 1 0 1 0 0 1 ) ( u u u u F + + ⋅ − ⋅ = o B, k α α α 1 , 1 0 0 0 , B B F u u u F ⋅ + − =

(

,1 ,0

)

2 , B B B F F F = − + p p₀₀ p p₁₁ p p₂₂ F_B,1 F_B,2 F_B,0 α α g F u F u u _i _B_i _i n i j j j i ext i i i i d k m m i − − − = +

∑

≠ =0 , , & & &

Equation

(22)

Kühnapfel / Çakmak _{Basic interactions: application of particle systems}_{Basic interactions: application of particle systems} 23.01.01 INT_P4 INT_P2 INT_P3 INT_P1 d d d' Bounding volumes Bounding volumes Range

Range testtest Distance test Distance test P_old P_new a' a n G S Flat Flat (nža)ž(nža‘)<0 v n v' v_N⋅ε_p v_T⋅(1-µ_p) Friction coefficient Friction coefficient µ_p Elasticity coefficient Elasticity coefficient ε_p

Particle System

•

•CreationCreation ofof newnew particlesparticles •

•Change of velocity Change of velocity and and directiondirection

Deformable Object

Object deformation after impact

v v v v F_ext = m⋅k⋅( 2 − '2) ⋅

Rigid Instruments

Deformable

_Deformable

Objects

G P_new P_old a' n Z (nža‘)<0 Organ Organ Pipe Pipe

(23)

H. Çakmak / Kü 23.01.01

Simulation of Suturing and Slings

• Suture material modelled as Spring-Mass-System

•

• Collision managementCollision management

(24)

H. Çakmak / Kü _{Suture Material Modelling}_{Suture Material Modelling} 23.01.01

Classification:

Material and

Absorbtion

Suture

types

dissolvable not dissolvable

Material absorbale not permanent permanent

anorganic — — Steel,Titanium:tZ=0

organic Kollagen, Catgut: tR =8d, tA=30d Silk, Yarn : tR=0, tZ=?-4 a —

synthetic Polyglykolsäure: tR=15d, tA=120d Polyamide: tR=0, tZ=1-2Ja Polyester: tZ=0

Polydioxanon: tR =35d, tA=180d Polypropylen: tR=0, tZ=1-5a

dissolvabledissolvable not not dissolvabledissolvable Material

Material absorbaleabsorbale not permanentnot permanent permanent permanent anorganic

anorganic — — Steel,Titanium:tZ=0

organic

organic Kollagen, Catgut: tR =8d, tA=30d Silk, Yarn : tR=0, tZ=?-4 a —

synthetic

synthetic Polyglykolsäure: tR=15d, tA=120d Polyamide: tR=0, tZ=1-2Ja Polyester: tZ=0

Polydioxanon: tR =35d, tA=180d Polypropylen: tR=0, tZ=1-5a

Modelling

Monofil

Monofil PseudoPseudo--monofilmonofil MultifilMultifil

Stiff

(25)

H. Çakmak / Kü 23.01.01

Self

-

collision of

suture

material

Hierarchical Collision Recognition

•

• Intersection of bounding spheres

Intersection of bounding spheres

•

• Intersection of cylinders (

Intersection of cylinders

(springs

springs)

)

Collision Response

•

• Definition of

Definition

of

active

active and passive element

and passive element

•

• Shifting

Shifting

of

of passive

passive cylinder

cylinder

iF

=

2

2 f / h

f / h

iR

=

pr

→

P

pr S_j=P Pnew VE_j h f f f VE_i,0 VE_(i+1), 0 Translation

(

)

pr P P_new = 1− iF ⋅ + iF ⋅

(26)

H. Çakmak / Kü _{Simulation of surgical suturing procedure}_{Simulation of surgical suturing procedure} 23.01.01

Collision Needle

-

Suture

- Organ models

Needle

-

Deformable object

:

• Register

• Register interacting object knots

interacting object knots

• Check force

• Check force treshold

treshold

• Find

• Find intersection

intersection

Ü

Definition

_Definition

of intersection as

_{of intersection as}

a constraint

_{a constraint}

Representation

Representation:

:

Barycentric coordinates

(a) (b)

(c) (d)

Suture

-

Deformable object

:

•

MutualMutual influence influence •

•Suture constrainedSuture constrained •

•Object deformation by suture motionObject deformation by suture motion

P1 P2 P3 P 2 2 1 1 0P P P P = w ₀ + w + w w₀ + w₁ + w₂ =1∧ w_i]0;1[

(27)

H. Çakmak / Kü 23.01.01

Sling Mechanism and Interaction

Sling mechanism

(28)

H. Çakmak / Kü _{Surgical Interactions with deformable objects}_{Surgical Interactions with deformable objects} 23.01.01

Modelling of

surgical

slings

Deformable effector

(a)

(a) Linkage

Linkage

(b) Dilatation

(c)

(c) Central knot

Central knot

for stabilisation

for

stabilisation

Functionality

:

Shortening of Sling

•

• Caculate

Caculate

new

knot positions

knot

positions

Shortening factor

sf

Simulation time

Simulation time step

step

∆

t

• Definition

(29)

H.K. Çakmak / Kü _{„Special effects“ for enhanced realism}_{„Special effects“ for enhanced realism} 23.01.01

Morphodynamics Simulation

T i m e T i m e Pulse Animation Pulse Animation Propagation of pressure Propagation of pressure

waves in an arterial tree

Animation of bleeding

after arterial injuries with

physiological correct data

e.g. Aorta: e.g. Aorta: Q = 83 ml/sQ = 83 ml/s r = 12 mm r = 12 mm v = 29 cm/s v = 29 cm/s

Stopping of bleeding with clips

Knowledge database

(30)

(31)

H. Çakmak / Kü 23.01.01

Arterial Bleeding

• Particle System Simulation

• Coupled with Pulse Simulation

• Application of clips to stop bleeding

• Accumulation of blood

• Parameters: Blood loss per vessel

Rendering settings

Visual Effects for Simulation Realism

(32)

(33)

Method

Texture

animation

Algorithm

•

Doubling of knot

P/P‘

after cut

•

Texture projection

Texture

projection

•

Time dependent texture scaling

Time

dependent texture scaling

Feature

Definition

of supply of blood at

each object knot

Application

Ü

Styptic

_Styptic

POR-8

_POR-8

injection

_injection

Ü

Coagulation

_Coagulation

stopping of bleeding

_{stopping of bleeding}

Ü

Irrigation

_Irrigation

of wound

_{of wound}

:

_:

Increase of texture transparency

Videoclip

„Special effects“ for enhanced realism: surface bleeding

(34)

H.K. Çakmak / Kü 23.01.01

Irrigation and Suction

• Implementation of Particle Systems

• Motion blurred particle rendering

• Tissue deformation: Particle impact

• Fluid accumulation • Fluid accumulation • Effects: Splashing • Effects: Splashing Reflection Reflection Ripples Ripples

Surgical Interactions with deformable objects (enhanced Realism)

(35)

H.K. Çakmak / Kü _{Simulation of Irrigation and Suction}_{Simulation of Irrigation and Suction} 23.01.01

Hydromechanics

:

Continuity equation

v u u ⋅ ⋅ = ⋅ ⋅ = ⋅ = = π 2 π 2 r dt d r dt d A dt dV Q

Modelling of

water

-

jet

Visual Effects

Fluid accumulation

Wave effects

Air

Air bubbles

bubbles

Instrument

specification

:

Fa

Fa. AESCULAP. AESCULAP

Q

Q_max_max=3 l/min, =3 l/min, dd=5 mm=5 mm ÜÜ vvmaxmax=2.55 m/s=2.55 m/s

Videoclip

(36)

H. Çakmak / Kü 23.01.01

Steam / Smoke for Coagulation

• 3D-Procedural Textures

• Billboard-technique

• Animation:

• Animation: Texture shiftingTexture shifting Spline Key-Frame

Spline Key-Frame

Color / Transparency

• Steaming up of endoscopic lense

Visual Effects for Simulation Realism: Coagulation Steam / Smoke

(37)

Basis

3D-3D-

Turbulence function(Perlin):

Turbulence function

(Perlin):

Density distribution

Method

1: 3D-Textures (a)

Visualisation

:

3D-Texture Mapping

Animation:

Modification of colours

Modification of colours and

and

transparencies

Shifting of

Shifting of Texture

Texture coordinates

coordinates

Method

2: 2D-Textures (b)

Visualisation

:

Billboard

Billboard-

-Technique

Technique

Viewpoint dependent scaling

Animation:

Texture

circulation

Spline-

Spline-Key

Key-Frame

-Frame

Technique

Coordination

with

instrument

activation

Steaming

up

of endoscopic lense

v A y x z (b) k₀ k₁ k_n-1 tm-1 t₀ t₁ tm-2 ... ... ... (a)

(38)

Videoclip

(39)

Direct haptic Interaction with Voxel-Volumes

Haptic Interaction with Voxel-Volumes

Application:

• Training in interventinal radiology • Needle injections (spine disks)

Method(s):

• use CT data for tissue-stiffness • gray-value encoding

• DICOM interface

• use colour lookup-table (RGBA) for direct volume-rendering of

CT-dataset (3D texture emulation) • Phantom as haptic display

(40)

Result(s):

• Training System for Minimally Invasive Surgery with realistic User-Interface

• Methods and SW-Tools for realistic Modelling of deformable Objects

• Realtime Simulation of Surgical Interactions:

Grasp, Clip, Cut, Coagulation, Irrigation, Slings, Suturing

• Active deformable Objects: Organ motility

• Particle Systems for Fluid Simulation

Conclusions:

• VR-based Surgical Simulation Systems will become much more realistic in the future

• They will be integrated into multimedia teaching and training environments • All surgical disciplines will be covered