5.6 Details

5.6.2 ModelReducer

The partitioning groups all values with the same Series Instance UID as key, making it possible for the reducer to analyze the images in an examination and generate a model describing the physical relationship of the images. The reduce function is encapsulated by the ModelReducer class.

The model consists of three levels of abstraction. The DisplayUnit class represents an entire series of images. A DisplayUnit is composed of a set of PhysicalRegions that contain ViewRegions geometrically and/or temporally related to each other. Two images are defined as geometrically related if their orientations are parallel. A ViewRegion simply corresponds to one image. A ViewRegion only contains the id, position and timestamp of the corresponding image.

The model is generated by iterating through all the ImageData objects given to the reduce function, adding one ViewRegion at a time by checking if it can be put into an existing PhysicalRegion or if a new one needs to created and added to the DisplayUnit.

When all ViewRegions have been added, a BSON representation of the model is generated and written to the specified output MongoDB collection.

Generating the Data Model 2012-10-25

30 5.7 Validation

To validate if the generated model was correct or not, another Java application was created.

This was not a main requirement, but helped to demonstrate what the model generator accomplished. The visualizer starts by reverse-engineering the

DisplayUnit-PhysicalRegion-ViewRegion model created in the reducing stage of the model generator, by fetching all generated documents from MongoDB. It was then possible to create 3D geometry for a DisplayUnit, using the Java3D API12. Each PhysicalRegion is color coded to easily distinguish between them. The user can manually select which PhysicalRegions to display.

Figure 10: Screenshot of the DICOM model visualizer used for validating the generated model.

12 http://www.oracle.com/technetwork/java/javase/tech/index-jsp-138252.html, August 2012

Summary & Discussion 2012-10-25

31

6 Summary & Discussion

6.1 Summary

This thesis contained of two parts. In the first part ways of storing medical images in a distributed manner were investigated. The purpose with this was to explore ways to provide improved scalability and flexibility compared to the current storage structure. It boiled down to exploring two rather separate problems; storing the actual images, and storing the

medically relevant metadata for each image. For the latter, several “non-relational databases” (NoSQL) were investigated, while a set of distributed file systems were

considered for storing the actual images. A requirement for the distributed storage was to support some kind of batch processing, providing analytical capabilities that scales well as data sizes grow larger and gets spread among multiple nodes. Being the most popular framework for this purpose, Hadoop MapReduce was used successfully on all the investigated systems.

In the second part, an application utilizing the power of the Hadoop MapReduce framework was implemented. A stack of images are often used to describe a volume in the patient’s body or a change of a certain area over time. The task of the implemented application was to generate a model more closely related to this structure, by processing the metadata of all images in the database.

Apache Cassandra, HBase, MongoDB, HDFS and GlusterFS were the products investigated the most. After investigating the systems, there were not any clear winners. Each had their own set of strengths and weaknesses. In the simple architecture of Cassandra all nodes in the cluster have the same role, making it easy to scale by adding nodes in a live cluster.

Some of the greatest strengths of HBase are the high performance concerning reads, and the use of the distributed file system to make data persistent. However, the cluster topology includes several different roles, adding much complexity compared to the amount of data to be handled. As opposed to the previous two, MongoDB is a document-based store. It is very flexible concerning both how data is stored and how to setup the cluster. For smaller

databases it is enough to have a single database where data is replicated automatically among several nodes. The two file systems most investigated were HDFS and GlusterFS.

Although HDFS is simple and works well with MapReduce, it has a master node as a single-point-of-failure, and is not as general as GlusterFS.

In the current structure of the Sectra PACS, there is a short-term and a long-term archive.

For the short-term archive, performance is the most crucial factor to consider. Expandability of storage capacity is not a big issue since old data is moved after a certain time limit. A distributed storage would therefore not be a very good fit in this case, even if it would add flexibility through metadata storage and analytical capabilities through Hadoop MapReduce, in addition to using cheaper hardware. In the long-term archive however, minimizing the latencies is not as critical, and the amount of data is constantly growing. This makes it a good fit for replacing current SANs or object-databases etc.

Summary & Discussion 2012-10-25

32 6.2 Why NoSQL?

There are several reasons why a non-relational database fits well for storing DICOM data.

Since the different images contain different sets of DICOM tags, storing all tags for every image would result in a large schema and a sparse table for a RDBMS. This is one of the cases NoSQL databases are meant for. There is generally no need for a strict schema, and empty values are simply not stored, which is efficient for sparse tables.

NoSQL databases are generally more suited for a distributed setting than relational

databases. For large sites the data could exceed the size limit to fit on a single node, as they currently do. A failure would require manually recovering using a backup. MongoDB, Apache Cassandra and HBase all support distribution among multiple nodes, data replication, and automatic failover.

The biggest advantage with a RDBMS is the SQL language, enabling very efficient and extensive querying capabilities. The NoSQL databases generally only support rather simple queries, such as query by key. MongoDB provides good querying capabilities without the creation of indexes, making it a good replacement for a SQL database.

6.3 Short-term storage

In a production environment, the time it takes to randomly access data from the short-term archive greatly affects the overall performance of how responsive the whole system feels.

There are rather tough requirements for the latency of the archive. Typically, a latency of no more than 10ms during a load of around 100 I/O operations per second is desirable. In this aspect, a SAN is probably the preferred choice since the benchmark did not show very high performance for GlusterFS. On the other hand GlusterFS nodes can be accessed in parallel from the clients, which would potentially improve performance during heavy load.

The number of images stored in the short-term archive differs depending on how big the individual site is, but it is typically in the order of tens or hundreds of millions. Both a SAN and a GlusterFS cluster are more than capable of handling this amount of data. The

distributed alternative has the advantage of being cheaper. By using commodity hardware there is the additional advantage of having extra computational power that can be used for running MapReduce jobs for example. This significantly increases the flexibility of the system.

By having MongoDB running on the same machines as GlusterFS (or on an independent cluster), metadata could also be stored in a distributed way. At least in principle the MongoDB cluster could replace parts of the WISE database containing information about patients and their examinations among other things. There are several advantages with the distributed alternative. It supports scaling through automated sharding of data if there should be the need. In this case the amount of data is not that large (tens or hundreds of gigabytes) which means it could fit on a single node, but by having replicas of the database in

Summary & Discussion 2012-10-25

33

the cluster there is the possibility of balancing the load among the replicas to avoid

bottlenecks. During heavy load WISE might be a bottleneck in the current system structure.

As a conclusion it might be better to stick with a SAN for short-term storage since minimizing latency is of great importance. The data is moved from the short-term storage after a limited time, with the consequence that the amount of data stored is more or less constant. This again goes in favor of the SAN, since there will not be a need for continuously adding expensive hardware.

6.4 Long-term storage

If an examination is not in the short-term archive when it is requested, all images belonging to this examination is fetched from the long-term archive and put into the short-term archive again before it can be viewed by the physician who requested the examination.

The performance requirements are not as critical for the long-term storage, since images older than a certain amount of time are not accessed that often. This brings the question whether it is worth investing a lot of money in a SAN, when the performance benefits are not giving the same value back to the system performance.

The continuously expanding amount of data makes it a good fit for a distributed storage, since it is made for scaling easily. Adding a node to a live cluster is supported for GlusterFS.

Because of this and the fact that cheaper hardware compared to a SAN can be used, GlusterFS (or another DFS) is well suited for this type of storage.

There is a purpose for having a distributed database for the long-term storage as well, making it possible to do queries on what images that need to be fetched among other things. Some of this data is stored in the WISE database to be able to fetch old examinations.

It would be beneficial to use a distributed database for the long-term archive since the data size is constantly growing. In a sharded MongoDB collection queries are distributed among several nodes, increasing the performance which is important for certain queries that are often requested.

6.5 MapReduce

Organizations such as Google, Yahoo!, Facebook and many more are using MapReduce for many diverse use cases including distributed sort, web link graph traversal, log file statistics, document clustering and machine learning. The number of common use cases is

continuously growing [1]. Integrating Hadoop MapReduce with many different products and implementing the model generating job has given lots of positive experiences. The

framework provides a lot of flexibility, both considering the configuration and the usage of the API. Implementing jobs in other languages is supported through Hadoop Streaming, which further proves the possibilities.

Summary & Discussion 2012-10-25

34

It should perform at its best when working with large files, since one map task is run for each file which introduces an overhead. For smaller data sets, or a large set of small files, the gain would lie more in simplifying the batch processing, rather than increased performance.

6.6 Future work

A large part of this thesis was spent reading about the products and their features, leaving less time to evaluate them. The evaluated products were only tested in a virtual

environment, where the benchmark may not have been very accurate. It would be

interesting to use them in a more realistic cluster setting, getting more accurate data about read/write speeds.

To further compare to the current solution it would also be appropriate with a more detailed cost evaluation.

During discussions with several employees at Sectra, potential usages for NoSQL databases in other situations were considered. For example using simple key/value stores to cache data locally, potentially increasing performance of the PACS. As mentioned in Section

3.1.7.2, graph databases could also have their applications. Therefore it would be of interest to investigate these as well.

References 2012-10-25

35

7 References

[1] S. Tiwari. Professional NoSQL. Wrox; 1st edition, 2011. ISBN: 047094224X

[2] S. Ghemawat, H. Gobioff, S-T. Leung. The Google File System. In SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles, Pages 29-43, 2003.

http://research.google.com/archive/gfs.html, doi: 10.1145/945445.945450

[3] Chang, Dean, Ghemawat, Hsieh, Wallach, Burrows, Chandra, Fikes, Gruber. BigTable: A Distributed Storage System for Structured Data. In ACM Transactions on Computer Systems (TOCS), Volume 26 Issue 2, June 2008, Article No. 4.

http://research.google.com/archive/bigtable.html, doi: 10.1145/1365815.1365816 [4] J. Dean, S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In

Communications of the ACM - 50th anniversary issue: 1958 – 2008, Volume 51 Issue 1, January 2008, Pages 107-113. http://research.google.com/archive/mapreduce.html, doi: 10.1145/1327452.1327492

[5] A. Lakshman, P. Malik. Cassandra - A Decentralized Structured Storage System. In PODC '09 Proceedings of the 28th ACM symposium on Principles of distributed computing, Pages 5-5, 2009.

http://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf, doi: 10.1145/1582716.1582722

[6] R. Zuidhof, J. van der Til. Comparison of MapReduce Implementations. In Proceedings 8th Student Colloquium 2010-2011, Pages 60-65, 2011.

http://altmetis.eldoc.ub.rug.nl/FILES/root/2011/eightproceedings_sme/studcol2011.p df

[7] E. Jansema, J. Thijs. Comparison between NoSQL Distributed Storage. In Proceedings 8th Student Colloquium 2010-2011, Pages 119-123, 2011.

http://www.mendeley.com/research/comparison-between-nosql-distributed-storage-systems/

[8] R. Henricsson. MongoDB vs CouchDB performance, 2011,

http://www.bth.se/fou/cuppsats.nsf/774564d376efaa0cc1256cec0031533e/32737dee 280f07ddc12578b200454a24!OpenDocument (2012-09-04)

[9] I. Varley. No Relation: The Mixed Blessings of Non-Relational Databases, 2009, http://ianvarley.com/UT/MR/Varley_MastersReport_Full_2009-08-07.pdf (2012-09-04)

[10] J. Luciani. Cassandra File System Design, 2012,

http://www.datastax.com/dev/blog/cassandra-file-system-design (2012-09-04)

References 2012-10-25

36

[11] C. Robinson. Cassandra Query Language (CQL) v2.0 reference,

http://crlog.info/2011/09/17/cassandra-query-language-cql-v2-0-reference/

(2012-09-04)

[12] DataStax Enterprise, http://www.datastax.com/products/enterprise (2012-09-04) [13] Apache HBase Reference Guide - http://hbase.apache.org/book.html (2012-09-04) [14] MongoDB Manual - http://www.mongodb.org/display/DOCS/Manual (2012-09-04) [15] HDFS Architecture Guide -

hadoop.apache.org/common/docs/current/hdfs_design.html (2012-09-04) [16] GlusterFS Features -

http://www.gluster.org/community/documentation/index.php/GlusterFS_Features (2012-09-04)

[17] The DICOM Standard - http://medical.nema.org/standard.html (2012-09-04) [18] Giuseppe DeCandia, Hastorun, Jampani, Kakulapati, Lakshman, Pilchin,

Sivasubramanian, Vosshall, Vogels. Dynamo: Amazon’s Highly Available Key-value Store. In SOSP '07 Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, Pages 205-220, 2007.

http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf, doi: 10.1145/1294261.1294281

Appendix 2012-10-25

public static class UnitMapper extends

Mapper<Object, BSONObject, Text, ImageData> {

data.setType(ImageType.getTypeByModality(modality));

// Check if mandatory attributes exist if(!data.getType().checkValidity(value)) { return;

}

Appendix 2012-10-25

38

data.setImgUID((String) value.get("SOPInstanceUID"));

TimeValue t = data.getType().getTimeValue(value);

public static class ModelReducer extends

Reducer<Text, ImageData, Text, BSONObject> {

Appendix 2012-10-25

public static void main(String[] args) throws Exception { Configuration conf = new Configuration();

job.setJarByClass(ModelReducer.class);

job.setMapperClass(UnitMapper.class);

job.setReducerClass(ModelReducer.class);

job.setMapOutputKeyClass(Text.class);

job.setMapOutputValueClass(ImageData.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(BSONObject.class);

job.setInputFormatClass(MongoInputFormat.class);

Appendix 2012-10-25

40

job.setOutputFormatClass(MongoOutputFormat.class);

System.exit(job.waitForCompletion(true) ? 0 : 1);

} }

Appendix 2012-10-25

41 8.2 Abbreviations

There are many abbreviations used frequently in this thesis. Here follows a list that briefly explains most of them:

PACS – Picture Archiving and Communication System

DICOM - Digital Imaging and Communications in Medicine, standard for handling, storing, printing, and transmitting information in medical imaging. Used by Sectra among others.

NoSQL – Collective term for databases that are not relational in the traditional sense.

Includes key/value stores, document stores, wide column stores, object databases, graph databases and more.

DFS – Distributed File System, general term used frequently.

HDFS – Hadoop Distributed File System, open-source counterpart to GFS.

GFS – Google File System, described in [2].

CFS – Cassandra File System, a HDFS compatible file system built upon Apache Cassandra, used by DSE.

URI – Uniform Resource Identifier, a string used to identify a name or resource.

DSE – DataStax Enterprise, a commercial product based on Cassandra.

JSON – JavaScript Object Notation, a lightweight data-interchange format.

BSON – Binary JSON, a binary-encoded serialization of JSON documents.

POSIX – Portable Operating System Interface, standard for maintaining compatibility between operating systems.

På svenska

Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under en längre tid från publiceringsdatum under förutsättning att inga extra-ordinära omständigheter uppstår.

Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art.

Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart.

För ytterligare information om Linköping University Electronic Press se förlagets hemsida

http://www.ep.liu.se/

In English

The publishers will keep this document online on the Internet - or its possible replacement - for a considerable time from the date of publication barring exceptional circumstances.

The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose.

Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility.

According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement.

For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page:

http://www.ep.liu.se/

© Tobias Dahlberg

In document Institutionen för datavetenskap Department of Computer and Information Science (Page 39-0)