0
2
4
6
8
10
12
14
Error rates mean (%)
Figure 8.5: Comparison of average error rate and standard error of the mean of ASA, deep learning (Baseline), IDL, ADL.
IDL ADL DL approach 0 20 40 60 80 100 120
Total verification time (mintues)
(a) Iterative DL Active DL Counting method 0 2 4 6 8 10 12 14 16
Average of verification time (mintues)
(b)
Figure 8.6: a) Total verification time comparison between Iterative deep learning and Active deep learning, b) average verification and standard error of the mean across three folds of
DL (Baseline) IDL ADL DL approach 0 500 1000 1500 2000 2500 3000 3500 4000
Total verified images
(a) Iterative DL Active DL Counting method 0 100 200 300 400 500
Average of verified images
(b)
Figure 8.7: a) Total number of verified images for DL (baseline), IDL, and ADL, b) Average and standard error of the mean of number of verified images for IDL and ADL.
DL (Baseline) IDL ADL
DL approach 0 100 200 300 400 500 600 700
Total accepted images
(a) Iterative DL Active DL Counting method 0 20 40 60 80 100
Average of accepted images
(b)
Figure 8.8: a) Total number of accepted images for DL (baseline), IDL, and ADL, b) Average and standard error of the mean of number of accepted images for IDL and ADL.
Chapter 9: Conclusions and Future Work
9.1 Summary
In Chapter 1, a discussion and motivations for automating unbiased stereology cell counting
to avoid the shortcomings of the manual unbiased stereology approach was presented.
In Chapter 2, background on the main concepts and terminologies presented in this
dissertation was discussed, which included unbiased stereology, learning algorithms, deep
learning, and active learning.
In Chapter 3, a detailed review of deep learning-based cell detection and segmentation in
microscopic images was presented. The review divides deep learning methods in detecting
cells into two types: cell center-based detection and bounding box based detection. The
segmentation methods reviewed are divided into two categories: semantic segmentation and
instance segmentation.
In Chapter 4, I provided details of the data set of NeuN single stain sections of mice
brains microscopy images. Moreover, a ground truth generation approach and verification by
human approach are explained.
In Chapter 5, I presented a deep learning framework for automating cell counting in
NeuN single stained sections microscopy images using deep learning and unbiased stereology
counting rules. Additionally, we investigated the effectiveness on deep learning performance by
increasing the training data of a deep learning model using different augmentation approaches.
In Chapter 6, I discussed an iterative approach called iterative deep learning (IDL) to
masks followed by human-in-the-loop verification. This approach achieved good results, but
it requires human effort in verification of predicted masks especially for large unlabeled sets
of microscopy images.
In Chapter 7, I presented an active deep learning (ADL) approach to reduce the user time
for verification by adding an ensemble-based query approach. This approach provided the
most confident images to be verified by the user. The user decision is simply accept or reject.
In Chapter 8, an experimental study of two deep learning frameworks (libraries) repro-
duciblity is presented. This study investigated the effect of setting a seed for randomization
in training deep learning models. Moreover, the study provided a list of libraries that require
seeding prior to and during training of a deep learning model. The study showed reproducible
results using Pytorch. Moreover, we provided the iterative deep learning and active deep
learning performance when training and testing using Pytorch deep learning library, which
provides reproducible results.
9.2 Contributions
In Chapter 5, a deep learning method to segment and count cells based on unbiased
stereology is proposed. This method is the first deep learning application to automate
unbiased stereology. The ground truth for this method was generated using a previously
proposed algorithm (ASA), followed by human verification, which alleviates the burden of
creating ground truth (binary masks) segmentation manually, as described in Chapter 4.
While deep learning-based unbiased stereology provides higher results compared to ASA, its
time and human effort costs are less than manual unbiased stereology.
In Chapter 6, an iterative approach to leverage unlabeled data is proposed. The purpose
of this algorithm is to include more training instances (images) for the training set which
user verifies deep learning generated masks and accepts or rejects based on the match between
manual annotation and the generated masks. This approach increases the performance of
a deep learning model by adding new images to the training set; however, it requires the
user to verify all of the images available in the active set, which is time consuming for larger
unlabeled data set.
In Chapter 7, an active deep learning approach that aims to reduce verification human
time and effort when verifying deep learning generated masks of unlabeled data. This
approach works in a similar manner to the approach proposed in Chapter 6, except that
the user is given high confidence images/masks to verify. Querying high confidence masks
from unlabeled data is based on an ensemble of multiple models generated masks, where
the multiple models are saved during the training of a single model. The saved models are
referred to as snapshot models that capture the variations while training a single model. This
approach reduced the number of verified masks by a user, and thus the verification time and
effort are reduced.
Chapter 8, presents an experimental study of reproducibility of training and re-training of
a deep learning models seven times using two deep learning libraries: Keras (with Tensorflow
backend) and Pytorch. The deep learning architecture used for the experiment was U-Net [33]
to generate masks of cells on microscopy images, where the result evaluation was based on
unbiased stereology cell counting as discussed in Chapter 5, Section 5.3.3. Moreover, the
results show parameter settings of deep learning to reduce the randomization.
9.3 List of Accepted and Published Papers
• Alahmari, S. S., Goldgof, D., Hall, L., Phoulady, H. A., Patel, R. H., & Mouton, P.
R. (2019). Automated cell counts on tissue sections by deep learning and unbiased
• Alahmari, S., Goldgof, D., Hall, L., Dave, P., Phoulady, H. A., & Mouton, P. (2018,
December). Iterative deep learning based unbiased stereology with human-in-the-loop.
In 2018 17th IEEE International Conference on Machine Learning and Applications
(ICMLA) (pp. 665-670). IEEE.
• Alahmari, S. S., Goldgof, D., Hall, L. O., & Mouton, P. R. (2019, October). Automatic
Cell Counting using Active Deep Learning and Unbiased Stereology. In 2019 IEEE
International Conference on Systems, Man and Cybernetics (SMC) (pp. 1708-1713).
IEEE.
9.4 Future Works
The work presented in Chapters 5, 6, and 7 will be extended to different data sets
with multiple stains and cells types such as microglia. The future experiments will use the
following data sets: NeuN dual stain data set, Iba1 dual stain data set, Iba 1 single stain data
set. For dual stain data sets, we plan to use deconvolution approach to separate stains [188].
Some microscopy image examples of these data sets are shown in Figure 9.1.
Another future approach is to extend the work presented in Chapter 7, to allow the user
to outline and create masks for low confidence masks generated by a deep learning ensemble
approach. Since the user is creating the masks manually for selected images, we plan to
select K images to reduce the human effort. Additionally, we plan to use more confidence
measurements such as entropy of background and foreground pixel-wise across all snapshot
models.
The current work presented in this dissertation uses the EDF images of stacks, where
the EDF image is a synthetic single created from 10 stack frames. This approach may lead
to obscured cells. Therefore, we are planning to use the individual frames for segmentation
(a) NeuN single stain (b) NeuN dual stain
(c) Iba1 single stain (d) Iba1 dual stain
Figure 9.1: Examples of microscopy images from four data sets of microscopy images of mice brain stained sections. The green lines of the disector frame are the inclusion lines, whereas
the red lines are the exclusion lines.
Leveraging unlabeled data is of great interest nowadays due to the challenges associated
with getting manual ground truth and the advancements of image acquisition tools, which
makes the labeling (ground truth) process a bottleneck. Therefore, we are planning to apply
self-supervised deep learning [189] to leverage unlabeled data with minimum ground truth
effort, and to achieve better unbiased sterelogy performance.
In unbiased stereology, bounding-box localization of cell-based counting is an approach
that could alleviate the burden of counting overlapping cells and may require less post-
processing compared to segmentation based unbiased stereology. Therefore, we plan to use
localization-based unbiased stereology in the future. On the other hand, unbiased stereology
counting is based on stacks of images (3D); therefore, using stacks to perform the detection
and segmentation of cells using deep learning could improve the performance of unbiased
stereology cell counting.
Recently, a new type of neural network Sabour et al. called a Capsule Network [190] was
proposed. Capsule Networks replace max-pooling layers with convolutional layers with strides
and a dynamic routing algorithm [190]. Additionally, each piece of information at a single
neuron of the neural network is stored as a capsule vector rather than a scalar in CNN based
networks. Henceforth, more information is stored, such as orientation and the magnitude
of the spatial features. The dynamic routing algorithm [190] routes capsule vectors based
on their agreement to the next layer of the capsule network. This technique enables the
preservation of information and maintains information on the relationship of part-to-whole.
Since the relationship between cell, surrounding objects, and cytoplasm carries information,
Capsule Networks for microscopy image analysis will enable learning robust features that
take the relationship of nuclei and the surrounding tissue parts into account. SegCaps [45]
introduced deconvolution modules for Capsule Networks in a similar approach to U-Net [33]
to microscopy images analysis, including unbiased stereology, will have huge success and will
References
[1] M. J. West, Basic Stereology for Biologists and Neuroscientists. Cold Spring Harbor Laboratory Press, 2012.
[2] P. R. Mouton, Unbiased stereology: a concise guide. JHU Press, 2011.
[3] M. Burke, S. Zangenehpour, P. R. Mouton, and M. Ptito, “Knowing what counts: unbiased stereology in the non-human primate brain,” Journal of visualized experiments: JoVE, p. 27, 2009.
[4] M. West, L. Slomianka, and H. J. G. Gundersen, “Unbiased stereological estimation of the total number of neurons in the subdivisions of the rat hippocampus using the optical fractionator,” The Anatomical Record, vol. 231, no. 4, pp. 482–497, 1991.
[5] P. R. Mouton, H. A. Phoulady, D. Goldgof, L. O. Hall, M. Gordon, and D. Morgan, “Unbiased estimation of cell number using the automatic optical fractionator,” Journal
of chemical neuroanatomy, vol. 80, pp. A1–A8, 2017.
[6] J.-G. Lee, S. Jun, Y.-W. Cho, H. Lee, G. B. Kim, J. B. Seo, and N. Kim, “Deep learning in medical imaging: general overview,” Korean Journal of Radiology, vol. 18, no. 4, pp. 570–584, 2017.
[7] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” science, vol. 313, no. 5786, pp. 504–507, 2006.
[8] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807–814.
[9] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
[10] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167, 2015.
[11] Y. LeCun, K. Kavukcuoglu, and C. Farabet, “Convolutional networks and applications in vision,” in Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International
[12] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. S´anchez, “A survey on deep learning in medical image analysis,” Medical image analysis, vol. 42, pp. 60–88, 2017.
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
[14] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255.
[15] B. Settles, “Active learning literature survey,” University of Wisconsin-Madison De- partment of Computer Sciences, Tech. Rep., 2009.
[16] S. Tong and E. Chang, “Support vector machine active learning for image retrieval,” in Proceedings of the ninth ACM international conference on Multimedia. ACM, 2001, pp. 107–118.
[17] S. Tong and D. Koller, “Support vector machine active learning with applications to text classification,” Journal of machine learning research, vol. 2, no. Nov, pp. 45–66, 2001.
[18] Y. Liu, “Active learning with support vector machine applied to gene expression data for cancer classification,” Journal of chemical information and computer sciences, vol. 44, no. 6, pp. 1936–1941, 2004.
[19] A. Top, G. Hamarneh, and R. Abugharbieh, “Active learning for interactive 3d image segmentation,” in International Conference on Medical Image Computing and Computer- Assisted Intervention. Springer, 2011, pp. 603–610.
[20] C. Sommer, C. Straehle, U. Koethe, and F. A. Hamprecht, “Ilastik: Interactive learning and segmentation toolkit,” in 2011 IEEE international symposium on biomedical imaging: From nano to macro. IEEE, 2011, pp. 230–233.
[21] A. Vezhnevets, J. M. Buhmann, and V. Ferrari, “Active learning for semantic seg- mentation with expected change,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012, pp. 3162–3169.
[22] S. S. Alahmari, D. Goldgof, L. Hall, H. A. Phoulady, R. H. Patel, and P. R. Mouton, “Automated cell counts on tissue sections by deep learning and unbiased stereology,”
Journal of chemical neuroanatomy, vol. 96, pp. 94–101, 2019.
[23] D. Sterio, “The unbiased estimation of number and sizes of arbitrary particles using the disector,” Journal of microscopy, vol. 134, no. 2, pp. 127–136, 1984.
[24] R. Rubin, D. S. Strayer, E. Rubin et al., Rubin’s pathology: clinicopathologic foundations of medicine. Lippincott Williams & Wilkins, 2008.
[25] M. N. Gurcan, L. Boucheron, A. Can, A. Madabhushi, N. Rajpoot, and B. Yener, “Histopathological image analysis: A review,” IEEE reviews in biomedical engineering,
vol. 2, p. 147, 2009.
[26] S. Roy, A. kumar Jain, S. Lal, and J. Kini, “A study about color normalization methods for histopathology images,” Micron, vol. 114, pp. 42–61, 2018.
[27] A. C. Ruifrok, D. A. Johnston et al., “Quantification of histochemical staining by color deconvolution,” Analytical and quantitative cytology and histology, vol. 23, no. 4, pp. 291–299, 2001.
[28] A. Vahadane, T. Peng, A. Sethi, S. Albarqouni, L. Wang, M. Baust, K. Steiger, A. M. Schlitter, I. Esposito, and N. Navab, “Structure-preserving color normalization and sparse stain separation for histological images,” IEEE transactions on medical imaging, vol. 35, no. 8, pp. 1962–1971, 2016.
[29] T. M. Mitchell et al., “Machine learning. 1997,” Burr Ridge, IL: McGraw Hill, vol. 45, no. 37, pp. 870–877, 1997.
[30] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning. MIT press, 2016.
[31] E. Alpaydin, Introduction to machine learning, 2020.
[32] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015.
[33] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
[34] B. Settles, “Active learning,” Synthesis Lectures on Artificial Intelligence and Machine Learning, vol. 6, no. 1, pp. 1–114, 2012.
[35] L. E. Atlas, D. A. Cohn, and R. E. Ladner, “Training connectionist networks with queries and selective sampling,” in Advances in neural information processing systems, 1990, pp. 566–573.
[36] D. Cohn, L. Atlas, and R. Ladner, “Improving generalization with active learning,” Machine learning, vol. 15, no. 2, pp. 201–221, 1994.
[37] D. D. Lewis and W. A. Gale, “A sequential algorithm for training text classifiers,” in SIGIR’94. Springer, 1994, pp. 3–12.
[38] M. N. Wernick, Y. Yang, J. G. Brankov, G. Yourganov, and S. C. Strother, “Machine learning in medical imaging,” IEEE signal processing magazine, vol. 27, no. 4, pp. 25–38, 2010.
[39] S. U. Akram, J. Kannala, L. Eklund, and J. Heikkil¨a, “Cell segmentation proposal network for microscopy image analysis,” in Deep Learning and Data Labeling for Medical Applications. Springer, 2016, pp. 21–29.
[40] V. Grau, A. Mewes, M. Alcaniz, R. Kikinis, and S. K. Warfield, “Improved watershed transform for medical image segmentation using prior information,” IEEE transactions on medical imaging, vol. 23, no. 4, pp. 447–458, 2004.
[41] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9.
[42] A. Izbassarova, A. Duisembay, and A. P. James, “Speech recognition application using deep learning neural network,” in Deep Learning Classifiers with Memristive Networks. Springer, 2020, pp. 69–79.
[43] Y.-H. Tu, I. Tashev, S. Zarar, and C.-H. Lee, “A hybrid approach to combining conventional and deep learning techniques for single-channel speech enhancement and recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 2531–2535.
[44] L. Deng, D. Yu et al., “Deep learning: methods and applications,” Foundations and Trends® in Signal Processing, vol. 7, no. 3–4, pp. 197–387, 2014.
[45] R. LaLonde and U. Bagci, “Capsules for object segmentation,” arXiv preprint arXiv:1804.04241, 2018.
[46] J. Ker, L. Wang, J. Rao, and T. Lim, “Deep learning applications in medical image analysis,” Ieee Access, vol. 6, pp. 9375–9389, 2017.
[47] F. Xing, Y. Xie, H. Su, F. Liu, and L. Yang, “Deep learning in microscopy image analysis: A survey,” IEEE transactions on neural networks and learning systems, vol. 29, no. 10, pp. 4550–4568, 2017.
[48] A. Savitzky and M. J. Golay, “Smoothing and differentiation of data by simplified least squares procedures.” Analytical chemistry, vol. 36, no. 8, pp. 1627–1639, 1964.
[49] J. Xu, L. Xiang, Q. Liu, H. Gilmore, J. Wu, J. Tang, and A. Madabhushi, “Stacked sparse autoencoder (ssae) for nuclei detection on breast cancer histopathology images,” IEEE transactions on medical imaging, vol. 35, no. 1, pp. 119–130, 2016.
[50] H. Fatakdawala, J. Xu, A. Basavanhally, G. Bhanot, S. Ganesan, M. Feldman, J. E. Tomaszewski, and A. Madabhushi, “Expectation–maximization-driven geodesic active contour with overlap resolution (emagacor): Application to lymphocyte segmentation on breast cancer histopathology,” IEEE Transactions on Biomedical Engineering, vol. 57, no. 7, pp. 1676–1689, 2010.
[51] H. Chang, J. Han, A. Borowsky, L. Loss, J. W. Gray, P. T. Spellman, and B. Parvin, “Invariant delineation of nuclear architecture in glioblastoma multiforme for clinical
and molecular association,” IEEE transactions on medical imaging, vol. 32, no. 4, pp. 670–682, 2012.
[52] R. Mufidah, I. Wasito, N. Hanifah, M. Faturrahman, and F. D. Ghaisani, “Automatic nucleus detection of pap smear images using stacked sparse autoencoder (ssae),” in Proceedings of the International Conference on Algorithms, Computing and Systems.