The introduction of this section is paraphrased from a related RC1 grant application written principally by Dr. Julian Rosenman and submitted through UNC Hospital’s Department of Radiation Oncology.
5.3.1
Clinical Goals
There are two main ways of determining the anatomic extent of a tumor: 3D imaging (e.g., CT, MRI, or PET) and direct visualization via endoscopy. In the head and neck, nasopharyngoscopy is a minimally invasive diagnostic procedure in which a small camera called an endoscope is inserted into the patient’s nasal passage and passed down through the pharynx. An attached cable serves to control the position and orientation of the camera, to provide light at the target via optical fiber, and to transmit a video signal back to a monitor. The endoscopic rig may include additional instruments such as a biopsy needle or pincer for collecting tissue samples at a target site.
Computed visualizations of 3D images and direct endoscopic evaluations are both particularly important when tumors are entangled with multiple critical normal tissues. Nasopharyngoscopy enables a clinician to see the mucosal surfaces of the cavities but not the internal tissues. 3D imaging provides information about the deep infiltration of the tumor, but it does not provide a visualization of the mucosal surfaces. So, for example, a patient with a submucosal abnormality on CT may have normal-appearing mucosa. Therefore the physician may require additional guidance from a 3D planning image to direct a biopsy. Or conversely, a CT may give incomplete information about the extent of a tumor's infiltration into the submucosal region. Therefore the physician may require additional guidance from direct visualization to estimate a target region for treatment planning. However, there is presently no easy way to visually register the information from these two assessment tools with each other beyond simply displaying the probe’s world position on a slice from a CT.
Virtual endoscopy has become accepted standard of care for certain clinical procedures such as colonoscopy (as early as the 1990s, see (Robb 1996), (Nain D. 2001)), but simulated views (Fig. 140 left) lack color information and cannot be used to guide actual biopsy. Real views
(Fig. 140 right), by contrast, obviously lack targeting information from any planning that has been done.
MGR methods can be used to create hybrid views from simulated and real views. There are two proposed applications for MGR in this domain. The first is to provide online guidance for the biopsy by integrating hidden features identified in 3D imaging directly into the endoscopic view. The second is the complementary task of integrating video from the endoscopy back into the 3D rendering to enhance open field of view virtual endoscopy with actual endoscopic images. This vignette could serve roles both for online guidance by indicating the probe position relative to the target region in 3D and for offline review. Taken together, these two vignettes would provide a powerful update to the standard guidance method of projecting the probe position onto the slice of a planning CT.
These techniques are expected to be of particular interest in virtual nasopharyngoscopy, which is an open area with little active research due at least in part to the large number of both discernible and inferred (lymph levels) critical anatomic structures and their complex spatial interrelationships. The mockup figures in the next two sections are accordingly taken from that domain.
5.3.2 Online Biopsy Guidance
Current methods for providing 3D visualizations suitable for augmenting endoscopic views are either limited to only a few poorly defined surfaces (i.e., the surface-only virtual colonoscopy model), or they take months to prepare and so are not applicable to the treatment of an individual patient (i.e., the VoxelMan model).
Fig. 140 Left, virtual
nasopharyngoscopy and right, corresponding image from real procedure.
The goal of this vignette is to augment the endoscopic view with information about the patient’s anatomy in the hidden “beyond-the- wall” region relative to the camera.
Given a partial segmentation of a planning image, MGR can synthesize overlays for the camera video feed to show both target objects discernible in the planning images and other clinically relevant objects and regions implied by prior anatomic knowledge.
The mgrView rendering engine runs fast enough to integrate this extra data in real time into the endoscopic view and overlay an annotated textbook-like “guided tour” of the nearby hidden features as the endoscope is advanced or endoscopic surgical instruments are utilized. It is expected that using this hybrid annotated view will improve diagnostic and therapeutic outcomes for endoscopy and endoscopic surgery for patients with head and neck cancer, in particular.
Method Outline
1. Collect a planning image, segment important objects and assign textures similarly to the head and neck project shown throughout chapter three, Model Guided Appearance for Medical Images.
Fig. 141 A mockup of an mgrView “guided tour” 2D endoscopic display showing a sample scope view embedded in a 3D planning image with target and nearby “beyond-the- wall” structures overlaid. Fig. 142 shows the complementary 3D view.
MGR Methods Required Combining 2D and 3D images
Regional volume textures
2. Register the endoscope position and orientation to the CT image and render an MGR virtual view from the point of view of a corresponding simulated camera.
3. Combine the views according to various levels of virtualization, e.g., camera only, camera + labels, camera + hidden objects, or virtual only.
5.3.3 Enhanced Open Field of View Virtual
Endoscopy
Endoscopic guidance is frequently restricted to projecting the probe position onto a 2D slice of the planning image. This can confound the user’s ability to understand the spatial relationship between the probe and nearby clinically relevant objects.
The goals of this vignette are to provide an unobstructed indicator for the relative positions of the probe and beyond the wall structures for online guidance and to put the endoscopic images back into a 3D context for review.
Fig. 142 A mockup of an mgrView open field of view virtual endoscopy enhanced with photomapping and online
guidance information. The
probe position relative to a target region is shown in 3D based on online probe position measurements. Color images collected by the endoscope are dynamically overlaid onto the CT.
Addressing the first goal, mgrView can easily take online probe position data and render a camera proxy into an open field of view endoscopic vignette as discussed previously. This can provide much needed 3D context for the spatial relationship between the probe and any target regions. This mapping additionally gives MGR all the information that it needs to project the current probe image back onto the CT image according to the methods described in Chapter 3. This photo-map enhanced image can then be used retrospectively for a color-correct virtual endoscopy, of either the fly-along-the-tube or the open field of view type that enables viewing from any point of view the physician desires. In either case, the virtual view could be further augmented by the same “beyond the wall” structures discussed for the previous vignette.
Method Outline
1. Use the methods from section 4.2, Fast Importance Rendering, to focus a view on the endoscopic path (an “open field of view virtual endoscopy”)
2. Use online camera position information to and add geometry proxies for the camera and nearby clinical target regions to the scene
3. Use a virtual camera to project each frame into the planning CT coordinates and color the local CT data according to the endoscopic view using the methods described in section 3.3.3, Rendering From Planar Images
4. Collect the entire color volume for use in offline for color-correct virtual endoscopy