Thesis

Thesis: 1. Current encoder-decoder networks cannot accurately delineate boundaries with low contrast due to the fuzzy information in the encoder layers. Learning high-resolution semantic features is a potential solution. Semantic-guided encoder feature learning can en- dow these architectures with high-resolution semantic features and thus can well handle the blurry boundary delineation problems. 2. Adversarial learning for supervised models work as realistic regularization. In this learning schema, the quantitative performance gain does not well align with the visual perception improvement. Adversarial confidence learning could achieve a synchronous performance gain by utilizing the confidence information to enhance the design of the supervised model while retaining the realistic effect. More impor- tantly, adversarial confidence learning can provide a platform for building difficulty-aware attention mechanism and confidence-aware semi-supervised algorithm to address easy sam- ple dominance issue and lack of labeled data, respectively, for deep learning based medical image segmentation and synthesis.

The first research topic of this dissertation is about low-contrast boundary delineation. It analyzes the limitation of the widely used encoder-decoder architecture based networks for blurry boundary delineation problems, and then proposes the idea that high-resolution semantic features could address this problem. Accordingly, this dissertation proposes three algorithms to efficiently learn detail-aware semantic features so that blurry boundary delin- eation could be well addressed.

The second research topic of this dissertation is about how to align the quantitative performance gain with visual perception improvement with adversarial learning for medical image analysis, how to solve easy-sample dominance issue and how to alleviate lack of annotated label problem. By investigating the roles of discriminators in classic GANs and comparing them with those in supervised adversarial learning systems, this dissertation figures out that adversarial learning works as realistic regularization for supervised models. It further proposes adversarial confidence learning framework to simultaneously achieve

performance improvement for quantitative and qualitative metrics. This framework utilizes the dense confidence information to improve the supervised generator with balancing the adversarial learning for the generator. It can be suited to difficulty-aware mechanism for alleviating the easy sample dominance issue for medical image analysis. It can also be adjusted to solve the lack of labeled data problem by adapting a semi-supervised learning algorithm.

1. Specific Aim (Blurry boundary delineation). The currently widely used encoder- decoder architectures (UNet is a typical example) can improve the segmentation accuracy by a large margin compared to conventional methods. While the encoder- decoder architecture cannot well handle the blurry boundary delineation problem be- cause the information provided by the encoder layers is fuzzy itself and thus cannot accurately localize the boundaries. For multi-modal data, designing a multi-modal network by taking the complementary information from multiple modalities into con- sideration is a good choice to address the low-contrast issue. For single-modal data, learning high resolution and rich semantic encoder features is a reasonable solution to address this problem. Accordingly, high-resolution pathway and semantic-guided encoder feature learning could both lead to learn high-resolution semantic features in the lateral connection and thus well fight against the blurry boundary delineation. 2. Specific Aim (Adversarial confidence learning). Adversarial learning can be

utilized to enhance the training of the neural networks for medical image analysis. Direct application of adversarial learning could lead to visual perception improve- ment, while the quantitative performance cannot have a synchronous growth and even become worse with adversarial learning. The adversarial learning works a re- alistic regularization for networks by enforcing the generated image to follow the distribution of real data in a entire image manner. However, it breaks the sample- to-sample correspondence which is the basis of the conventional loss for objective functions, which could thus limit the quantitative performance. This problem could

be alleviated by exploiting the confidence information from the discriminator to de- sign a more powerful supervised generator.

(a) Specific Aim (Difficulty-aware attention mechanism for medical im- age analysis). In medical image analysis, easy-to-segment (easy-to-synthesize) sample dominance phenomenon often occurs due to the irregular distribution of some medical images which may be caused by the different abnormal degree of the lesion or the imaging factors, such as different vendor devices or imaging protocols. To address this problem, conventional methods usually apply more weights to the difficult regions. However, it is hard to dynamically recognize the difficult-regions and determine the weights. By adopting the discriminator to encode the concatenation of probability map of segmentation and original input, the designed fully convolutional confidence network can obtain structured diffi- cult regions and learn reasonable confidence information in a dynamic manner. (b) Specific Aim (Confidence-aware semi-supervised learning for medical image segmentation). Deep learning based medical image segmentation re- quires large scale of annotated data. However, labeled data in medical image segmentation is usually challenging to obtain. Semi-supervised learning is a common choice to take advantage of unlabeled medical data. However, it is hard to determine which is a good segmented sample for unlabeled data and conventional semi-supervised segmentation models usually include the entire samples which have been certified to be well segmented. It largely limits the power of semi-supervised models due to the poorly segmented regions in the entire samples could provide wrong training signals. The adversarial confidence learning could provide a real-time well-or-poorly-segmented determination mech- anism for unlabeled data. More importantly, since this mechanism can select the high-confidence segmented regions, the problem by using the entire images as training signals could be elegantly solved.

In order to support the thesis and the above two main specific aims, the detailed contri- butions of this dissertation include:

1. A multi-modal evolutionary UNet is proposed to address the problems of low-contrast multi-modal medical image segmentation. Multi-modal information is utilized to ad- dress the low-constrast issue for medical image segmentation and fusion strategy is further explored in the environment of multi-modal neural networks. Besides, a transformation module composed of convolutional operations is proposed to alleviate the channel information bias between encoder and decoder features; a fusion module is proposed to better fuse information from encoder and decoder layers; (Aim 1) 2. High-resolution encoder-decoder networks are proposed to better delineate the blurry

boundaries for medical images, in which, dilated residual module is specifically de- signed to replace the skip connection in encoder-decoder networks; (Aim 1)

3. Semantic-guided encoder feature learning is proposed to efficiently learn high-resolution semantic features to endow encoder-decoder networks the capacity of blurry bound- ary delineation; (Aim 1)

4. Contour-sensitive loss functions, including regression and soft classification, are ex- plored for better modeling the boundaries to more clearly recognize the boundaries; (Aim 1)

5. Extensive experiments on several large medical segmentation datasets (infant brain segmentation in MRI, public challenge and self-owned pelvic datasets in both MRI and CT) indicate that the proposed method can accurately delineate the blurry boundaries with outperforming many existing methods in these tasks; (Aim 1) 6. Deep learning based method is proposed for cross-modality medical image synthesis.

realistic images. Auto-context model is also explored for alleviating the long-range information dependency problem. (Aim 2)

7. Analysis is conducted to explore the roles of the discriminator in classic GANs and comparison is done with those roles in supervised adversarial learning systems. To- gether with further experiments, adversarial learning is certified to work as realistic regularization in supervised adversarial learning systems; (Aim 2)

8. Adversarial learning can be utilized as realistic regularization for supervised models. The problem of adversarial learning is proposed: visual perception improvement does not well align with quantitative performance gain with standard adversarial learning; (Aim 2)

9. Adversarial confidence learning framework is proposed to address the inconsistency of performance gain in different metrics. By adopting a fully convolutional adversar- ial network, dense confidence information can be utilized to better design the super- vised generator networks to improve the quantitative performance, in the meantime, the realistic regularization with adversarial learning is retained; (Aim 2)

10. Following adversarial confidence learning, difficulty-aware attention mechanism is proposed for medical image segmentation and synthesis, especially hard-to-segment samples and lesion medical image synthesis; (Aim 2)

11. Confidence-aware semi-supervised segmentation networks are proposed to adaptively recognize the well segmented samples and regions. It targets at including the well- segmented regions instead of the entire sample to dynamically increase the labeled set; (Aim 2)

12. Extensive experiments that are conducted on several datasets indicate the effec- tiveness of my proposed approaches, especially the adversarial confidence learning framework. (Aim 2)