Hazelcast Documentation
version 3.6-EA2
2
In-Memory Data Grid - Hazelcast | Documentation: version 3.6-EA2 Publication date Nov 26, 2015
Copyright © 2015 Hazelcast, Inc.
Permission to use, copy, modify and distribute this document for any purpose and without fee is hereby granted in perpetuity, provided that the above copyright notice and this paragraph appear in all copies.
Contents
1 Preface 19
2 What’s New in Hazelcast 3.6 21
2.1 Release Notes . . . 21 2.1.1 New Features . . . 21 2.1.2 Enhancements . . . 22 2.1.3 Fixes . . . 23 2.2 Upgrading Hazelcast . . . 25 2.2.1 Upgrading from 2.x . . . 25 2.2.2 Upgrading from 3.x . . . 27
2.3 Document Revision History . . . 27
3 Getting Started 29 3.1 Installation . . . 29
3.1.1 Hazelcast . . . 29
3.1.2 Hazelcast Enterprise . . . 29
3.2 Starting the Member and Client . . . 31
3.2.1 Deploying On Amazon EC2 . . . 32
3.3 Configuring Hazelcast . . . 32 3.4 Phoning Home . . . 33 4 Hazelcast Overview 35 4.1 Sharding in Hazelcast . . . 36 4.2 Hazelcast Topology . . . 36 4.3 Why Hazelcast? . . . 37 4.4 Data Partitioning . . . 38
4.4.1 How the Data is Partitioned . . . 40
4.4.2 Partition Table . . . 40
4.4.3 Repartitioning . . . 41
4.5 Use Cases . . . 41
4.6 Resources . . . 42 3
4 CONTENTS
5 Hazelcast Clusters 43
5.1 Discovering Cluster Members . . . 43
5.1.1 Discovering Members by Multicast . . . 43
5.1.2 Discovering Members by TCP . . . 44
5.1.3 Discovering Members within EC2 Cloud . . . 45
5.1.4 Discovering Members with jclouds . . . 46
5.2 Creating Cluster Groups . . . 49
6 Distributed Data Structures 51 6.1 Map . . . 52
6.1.1 Getting a Map and Putting an Entry . . . 52
6.1.2 Backing Up Maps . . . 55
6.1.3 Evicting Map Entries . . . 56
6.1.4 Setting In Memory Format . . . 60
6.1.5 Loading and Storing Persistent Data . . . 61
6.1.6 Creating Near Cache for Map . . . 66
6.1.7 Locking Maps . . . 68
6.1.8 Accessing Entry Statistics . . . 71
6.1.9 Listening to Map Events . . . 71
6.1.10 Adding Interceptors . . . 73
6.1.11 Preventing Out of Memory Exceptions . . . 77
6.2 Queue . . . 78
6.2.1 Getting a Queue and Putting Items . . . 78
6.2.2 Creating an Example Queue . . . 78
6.2.3 Setting a Bounded Queue . . . 80
6.2.4 Queueing with Persistent Datastore . . . 81
6.2.5 Configuring Queue . . . 82
6.3 MultiMap . . . 82
6.3.1 Getting a MultiMap and Putting an Entry . . . 83
6.3.2 Configuring MultiMap . . . 83
6.4 Set . . . 83
6.4.1 Getting a Set and Putting Items . . . 84
6.4.2 Listening for Set Item Events . . . 84
6.5 List . . . 85
6.5.1 Getting a List and Putting Items . . . 85
6.5.2 Listening for List Item Events . . . 86
6.6 Ringbuffer . . . 86
6.6.1 Getting a Ringbuffer and Reading Items . . . 86
6.6.2 Adding Items to a Ringbuffer . . . 87
CONTENTS 5
6.6.4 Configuring Ringbuffer Capacity . . . 87
6.6.5 Backing Up Ringbuffer . . . 88
6.6.6 Configuring Ringbuffer Time To Live . . . 88
6.6.7 Setting Ringbuffer Overflow Policy . . . 88
6.6.8 Configuring Ringbuffer In-Memory Format . . . 89
6.6.9 Adding Batched Items . . . 89
6.6.10 Reading Batched Items . . . 89
6.6.11 Using Async Methods . . . 90
6.6.12 Ringbuffer Configuration Examples . . . 90
6.7 Topic . . . 91
6.7.1 Getting a Topic and Publishing Messages . . . 91
6.7.2 Getting Topic Statistics . . . 92
6.7.3 Understanding Topic Behavior . . . 92
6.7.4 Configuring Topic . . . 93
6.8 Reliable Topic . . . 94
6.8.1 Sample Reliable ITopic Code . . . 94
6.8.2 Slow Consumers . . . 95
6.9 Lock . . . 95
6.9.1 Using Try-Catch Blocks with Locks . . . 95
6.9.2 Releasing Locks with tryLock Timeout . . . 96
6.9.3 Avoiding Waiting Threads with Lease Time . . . 96
6.9.4 Understanding Lock Behavior . . . 96
6.9.5 Synchronizing Threads with ICondition . . . 97
6.10 IAtomicLong . . . 98
6.10.1 Sending Functions to IAtomicLong . . . 98
6.10.2 Executing Functions on IAtomicLong . . . 98
6.10.3 Reasons to Use Functions with IAtomic . . . 99
6.11 ISemaphore . . . 99
6.11.1 Controlling Thread Counts with Semaphore Permits . . . 99
6.11.2 Example Semaphore Code . . . 100
6.11.3 Example Semaphore Configuration . . . 101
6.12 IAtomicReference . . . 101
6.12.1 Sending Functions to IAtomicReference . . . 101
6.12.2 Using IAtomicReference . . . 102
6.13 ICountDownLatch . . . 102
6.13.1 Gate-Keeping Concurrent Activities . . . 102
6.13.2 Recovering From Failure . . . 103
6.13.3 Using ICountDownLatch . . . 103
6 CONTENTS
6.14.1 Generating Cluster-Wide IDs . . . 103
6.14.2 Unique IDs and Duplicate IDs . . . 104
6.15 Replicated Map . . . 104
6.15.1 Replicating Instead of Partitioning . . . 104
6.15.2 Example Replicated Map Code . . . 105
6.15.3 Considerations for Replicated Map . . . 105
6.15.4 Configuration Design for Replicated Map . . . 105
6.15.5 Configuring Replicated Map . . . 106
6.15.6 Using EntryListener on Replicated Map . . . 107
7 Distributed Events 109 7.1 Event Listeners for Hazelcast Members . . . 109
7.1.1 Listening for Member Events . . . 109
7.1.2 Listening for Distributed Object Events . . . 110
7.1.3 Listening for Migration Events . . . 111
7.1.4 Listening for Partition Lost Events . . . 111
7.1.5 Listening for Lifecycle Events . . . 112
7.1.6 Listening for Item Events . . . 112
7.1.7 Listening for Topic Messages . . . 113
7.1.8 Listening for Clients . . . 113
7.2 Event Listeners for Hazelcast Clients . . . 114
7.3 Global Event Configuration . . . 114
8 Distributed Computing 115 8.1 Executor Service . . . 115
8.1.1 Implementing a Callable Task . . . 115
8.1.2 Implementing a Runnable Task . . . 117
8.1.3 Configuring Executor Threads . . . 118
8.1.4 Scaling The Executor Service . . . 118
8.1.5 Executing Code in the Cluster . . . 118
8.1.6 Canceling an Executing Task . . . 119
8.1.7 Callback When Task Completes . . . 120
8.1.8 Selecting Members for Task Execution . . . 122
8.2 Entry Processor . . . 122
8.2.1 Performing Fast In-Memory Map Operations . . . 122
8.2.2 Creating an Entry Processor . . . 124
CONTENTS 7
9 Distributed Query 127
9.1 How Distributed Query Works . . . 127
9.1.1 Employee Map Query Example . . . 127
9.1.2 Querying with Criteria API . . . 128
9.1.3 Querying with SQL . . . 130
9.1.4 Filtering with Paging Predicates . . . 131
9.1.5 Indexing Queries . . . 131
9.1.6 Configuring Query Thread Pool . . . 132
9.2 Querying in Collections and Arrays . . . 133
9.2.1 Indexing in Collections and Arrays . . . 133
9.2.2 Corner cases . . . 134
9.3 Custom Attributes . . . 134
9.3.1 Implementing a ValueExtractor . . . 134
9.3.2 Extraction Arguments . . . 136
9.3.3 Configuring a Custom Attribute Programmatically . . . 136
9.3.4 Configuring a Custom Attribute Declaratively . . . 136
9.3.5 Indexing Custom Attributes . . . 136
9.4 MapReduce . . . 137
9.4.1 Understanding MapReduce . . . 137
9.4.2 Using the MapReduce API . . . 139
9.4.3 Hazelcast MapReduce Architecture . . . 146
9.5 Aggregators . . . 149
9.5.1 Aggregations Basics . . . 149
9.5.2 Using the Aggregations API . . . 150
9.5.3 Aggregations Examples . . . 154
9.5.4 Implementing Aggregations . . . 157
9.6 Continuous Query . . . 158
9.6.1 Listening to Predicate Map Entries . . . 158
9.7 Continuous Query Cache . . . 160
9.7.1 Keeping Query Results Local and Ready . . . 160
9.7.2 Accessing Continuous Query from Server Side . . . 160
9.7.3 Accessing Continuous Query from Client Side . . . 161
9.7.4 Features of Continuous Query Cache . . . 161
10 Transactions 163 10.1 Creating a Transaction Interface . . . 163
10.1.1 Queue/Set/List vs. Map/Multimap . . . 164
10.1.2 ONE_PHASE vs. TWO_PHASE . . . 164
10.2 Providing XA Transactions . . . 165
8 CONTENTS
10.3.1 Sample Code for J2EE Integration . . . 166
10.3.2 Configuring Resource Adapter . . . 166
10.3.3 Configuring a Glassfish v3 Web Application . . . 166
10.3.4 Configuring a JBoss AS 5 Web Application . . . 167
10.3.5 Configuring a JBoss AS 7 / EAP 6 Web Application . . . 168
11 Hazelcast JCache 171 11.1 JCache Overview . . . 171
11.2 JCache Setup and Configuration . . . 171
11.2.1 Setting up Your Application . . . 171
11.2.2 Example JCache Application . . . 173
11.2.3 Configuring for JCache . . . 174
11.3 JCache Providers . . . 176
11.3.1 Configuring JCache Provider . . . 176
11.3.2 Configuring JCache with Client Provider . . . 177
11.3.3 Configuring JCache with Server Provider . . . 177
11.4 JCache API . . . 177
11.4.1 JCache API Application Example . . . 177
11.4.2 JCache Base Classes . . . 179
11.4.3 Implementing Factory and FactoryBuilder . . . 180
11.4.4 Implementing CacheLoader . . . 180
11.4.5 CacheWriter . . . 181
11.4.6 Implementing EntryProcessor . . . 183
11.4.7 CacheEntryListener . . . 184
11.4.8 ExpirePolicy . . . 185
11.5 Hazelcast JCache Extension - ICache . . . 185
11.5.1 Scoping to Join Clusters . . . 186
11.5.2 Namespacing . . . 189
11.5.3 Retrieving an ICache Instance . . . 189
11.5.4 ICache Configuration . . . 189
11.5.5 ICache Async Methods . . . 190
11.5.6 Defining a Custom ExpiryPolicy . . . 192
11.5.7 JCache Eviction . . . 193
11.5.8 JCache Near Cache . . . 196
11.5.9 ICache Convenience Methods . . . 199
11.5.10 Implementing BackupAwareEntryProcessor . . . 199
11.5.11 ICache Partition Lost Listener . . . 200
11.5.12 JCache Split-Brain . . . 201
CONTENTS 9
12 Integrated Clustering 205
12.1 Hibernate Second Level Cache . . . 205
12.1.1 Sample Code for Hibernate . . . 205
12.1.2 Supported Hibernate Versions . . . 205
12.1.3 Configuring Hibernate for Hazelcast . . . 205
12.1.4 Configuring Hazelcast for Hibernate . . . 207
12.1.5 Setting P2P (Peer-to-Peer) for Hibernate . . . 208
12.1.6 Setting Client/Server for Hibernate . . . 208
12.1.7 Configuring Cache Concurrency Strategy . . . 209
12.1.8 Advanced Settings . . . 209
12.2 Web Session Replication . . . 209
12.2.1 Filter Based Web Session Replication . . . 210
12.2.2 Supporting Spring Security . . . 213
12.2.3 Tomcat Based Web Session Replication . . . 215
12.2.4 Jetty Based Web Session Replication . . . 218
12.3 Spring Integration . . . 222
12.3.1 Supported Versions . . . 222
12.3.2 Configuring Spring . . . 222
12.3.3 Enabling SpringAware Objects . . . 225
12.3.4 Adding Caching to Spring . . . 228
12.3.5 Configuring Hibernate Second Level Cache . . . 230
12.3.6 Best Practices . . . 230
13 Storage 233 13.1 High-Density Memory Store . . . 233
13.1.1 Configuring High-Density Memory Store . . . 233
13.2 Sizing Practices . . . 234
13.3 Hot Restart Store . . . 234
13.3.1 Hot Restart Store Overview . . . 235
13.3.2 Configuring Hot Restart . . . 235
13.3.3 Hot Restart and IP Address-Port . . . 237
13.3.4 Hot Restart Store Design Details . . . 237
13.3.5 Concurrent, Incremental, Generational GC . . . 237
14 Hazelcast Java Client 239 14.1 Hazelcast Clients Feature Comparison . . . 239
14.2 Java Client Overview . . . 240
14.2.1 Including Dependencies for Java Clients . . . 240
14.2.2 Getting Started with Client API . . . 241
10 CONTENTS
14.2.4 Handling Failures . . . 241
14.2.5 Using Supported Distributed Data Structures . . . 242
14.2.6 Using Client Services . . . 243
14.2.7 Client Listeners . . . 244
14.2.8 Client Transactions . . . 245
14.3 Configuring Java Client . . . 245
14.3.1 Configuring Client Network . . . 245
14.3.2 Configuring Client Load Balancer . . . 251
14.3.3 Configuring Client Near Cache . . . 251
14.3.4 Client Group Configuration . . . 252
14.3.5 Client Security Configuration . . . 252
14.3.6 Client Serialization Configuration . . . 252
14.3.7 Configuring Client Listeners . . . 252
14.3.8 ExecutorPoolSize . . . 253
14.3.9 ClassLoader . . . 253
14.4 Client System Properties . . . 253
14.5 Sample Codes for Client . . . 253
15 Other Client Implementations 255 15.1 C++ Client . . . 255
15.1.1 Setting Up C++ Client . . . 255
15.1.2 Installing C++ Client . . . 256
15.1.3 C++ Client Code Examples . . . 256
15.2 .NET Client . . . 260
15.2.1 Configuring .NET Client . . . 263
15.2.2 Starting .NET Client . . . 263
15.3 REST Client . . . 263
15.3.1 REST Client GET/POST/DELETE Examples . . . 264
15.3.2 Checking the Status of the Cluster for REST Client . . . 266
15.4 Memcache Client . . . 267
15.4.1 Memcache Client Code Examples . . . 267
15.4.2 Unsupported Operations for Memcache . . . 268
16 Serialization 269 16.1 Serialization Interface Types . . . 269
16.2 Comparing Serialization Interfaces . . . 270
16.3 Implementing Java Serializable and Externalizable . . . 270
16.3.1 Implementing Java Externalizable . . . 270
16.4 Implementing DataSerializable . . . 271
CONTENTS 11
16.5 Implementing Portable Serialization . . . 274
16.5.1 Portable Serialization Example Code . . . 275
16.5.2 Registering the Portable Factory . . . 276
16.5.3 Versioning for Portable Serialization . . . 276
16.5.4 Null Portable Serialization . . . 277
16.5.5 DistributedObject Serialization . . . 277 16.6 Custom Serialization . . . 278 16.6.1 Implementing StreamSerializer . . . 278 16.6.2 Implementing ByteArraySerializer . . . 280 16.7 Implementing HazelcastInstanceAware . . . 281 17 Management 283 17.1 Getting Member Statistics from Distributed Data Structures . . . 283
17.1.1 Map Statistics . . . 283
17.1.2 Multimap Statistics . . . 286
17.1.3 Queue Statistics . . . 289
17.1.4 Topic Statistics . . . 290
17.1.5 Executor Statistics . . . 291
17.2 JMX API per Node . . . 292
17.3 Monitoring with JMX . . . 298
17.3.1 MBean Naming for Hazelcast Data Structures . . . 299
17.3.2 Connecting to JMX Agent . . . 299
17.4 Cluster Utilities . . . 299
17.4.1 Getting Member Events and Member Sets . . . 299
17.4.2 Managing Cluster and Member States . . . 300
17.4.3 Enabling Lite Members . . . 301
17.4.4 Defining Member Attributes . . . 302
17.4.5 Safety Checking Cluster Members . . . 303
17.4.6 Defining a Cluster Quorum . . . 304
17.5 Management Center . . . 307
17.5.1 Installing Management Center . . . 307
17.5.2 Management Center Tools . . . 307
17.5.3 Management Center Home Page . . . 311
17.5.4 Monitoring Caches . . . 313
17.5.5 Managing Maps . . . 314
17.5.6 Monitoring Replicated Maps . . . 318
17.5.7 Monitoring Queues . . . 319
17.5.8 Monitoring Topics . . . 321
17.5.9 Monitoring MultiMaps . . . 321
12 CONTENTS
17.5.11 Monitoring WAN Replication . . . 323
17.5.12 Monitoring Members . . . 324
17.5.13 Scripting . . . 326
17.5.14 Executing Console Commands . . . 328
17.5.15 Creating Alerts . . . 328
17.5.16 Administering Management Center . . . 333
17.5.17 Hot Restart . . . 334
17.5.18 Checking Past Status with Time Travel . . . 335
17.5.19 Management Center Documentation . . . 337
17.5.20 Suggested Heap Size . . . 337
17.6 Clustered JMX via Management Center . . . 337
17.6.1 Configuring Clustered JMX . . . 338
17.6.2 Clustered JMX API . . . 338
17.6.3 Integrating with New Relic . . . 343
17.6.4 Integrating with AppDynamics . . . 344
17.7 Clustered REST via Management Center . . . 344
17.7.1 Enabling Clustered REST . . . 344
17.7.2 Clustered REST API Root . . . 344
17.7.3 Clusters Resource . . . 345 17.7.4 Cluster Resource . . . 345 17.7.5 Members Resource . . . 345 17.7.6 Member Resource . . . 346 17.7.7 Clients Resource . . . 349 17.7.8 Maps Resource . . . 349 17.7.9 MultiMaps Resource . . . 350 17.7.10 Queues Resource . . . 351 17.7.11 Topics Resource . . . 352 17.7.12 Executors Resource . . . 353 18 Security 355 18.1 Enabling Security for Hazelcast Enterprise . . . 355
18.2 Socket Interceptor . . . 355 18.3 Security Interceptor . . . 356 18.4 Encryption . . . 357 18.5 SSL . . . 358 18.6 Credentials . . . 359 18.7 ClusterLoginModule . . . 360 18.7.1 Enterprise Integration . . . 361
18.8 Cluster Member Security . . . 362
CONTENTS 13 18.9.1 Authentication . . . 363 18.9.2 Authorization . . . 364 18.9.3 Permissions . . . 365 19 Performance 369 19.1 Data Affinity . . . 369 19.2 Back Pressure . . . 372 19.3 Threading Model . . . 373 19.3.1 I/O Threading . . . 373 19.3.2 Event Threading . . . 374 19.3.3 IExecutor Threading . . . 374 19.3.4 Operation Threading . . . 374 19.4 SlowOperationDetector . . . 376
19.4.1 Logging of Slow Operations . . . 377
19.4.2 Purging of Slow Operation Logs . . . 377
19.5 Hazelcast Performance on AWS . . . 377
19.5.1 Selecting EC2 Instance Type . . . 377
19.5.2 Dealing with Network Latency . . . 378
19.5.3 Selecting Virtualization . . . 378
20 Hazelcast Simulator 379 20.1 Key Concepts . . . 379
20.2 Installing Simulator . . . 380
20.2.1 Firewall Settings . . . 380
20.2.2 Setting Up the Local Machine (Coordinator) . . . 380
20.2.3 Setting Up the Remote Machines (Agents, Workers) . . . 382
20.2.4 Setting Up the Public/Private Key Pair . . . 382
20.3 Setting Up For Amazon EC2 . . . 382
20.4 Setting Up For Google Compute Engine . . . 383
20.5 Setting Up Machines Manually . . . 383
20.6 Executing a Simulator Test . . . 384
20.6.1 Creating and Editing Properties File . . . 384
20.6.2 Running the Test . . . 385
20.6.3 Running the Test with a Script . . . 389
20.6.4 Using Maven Archetypes . . . 390
20.7 Provisioner . . . 390
20.7.1 Accessing the Provisioned Machine . . . 391
20.8 Coordinator . . . 391
20.8.1 Controlling Hazelcast Declarative Configuration . . . 392
14 CONTENTS
20.8.3 Controlling Client And Workers . . . 392
20.9 Communicator . . . 392
20.9.1 Example . . . 393
20.9.2 Message Types . . . 393
20.9.3 Message Addressing . . . 393
20.10Simulator.Properties File Description . . . 394
20.11Performance and Benchmarking . . . 395
21 WAN 397 21.1 WAN Replication . . . 397
21.1.1 Configuring WAN Replication . . . 397
21.1.2 WAN Replication Additional Information . . . 398
21.2 Enterprise WAN Replication . . . 399
21.2.1 Replication implementations . . . 399
21.2.2 WAN Replication Batch Size . . . 399
21.2.3 WAN Replication Batch Frequency . . . 400
21.2.4 WAN Replication Operation Timeout . . . 400
21.2.5 WAN Replication Queue Capacity . . . 400
21.2.6 WAN Replication Event Filtering API . . . 401
21.2.7 WAN Replication Acknowledge Types . . . 402
21.2.8 Enterprise WAN Replication Additional Information . . . 403
22 OSGI 405 22.1 OSGI Support . . . 405
22.2 API . . . 405
22.3 Configuring Hazelcast OSGI Support . . . 405
22.4 Design . . . 406
22.5 Using Hazelcast OSGI Service . . . 407
22.5.1 Getting Hazelcast OSGI Service Instances . . . 407
22.5.2 Managing and Using Hazelcast instances . . . 407
23 Hazelcast Configuration 409 23.1 Configuration Overview . . . 409
23.2 Using Wildcard . . . 411
23.3 Using Variables . . . 412
23.4 Composing Declarative Configuration . . . 412
23.5 Network Configuration . . . 414
23.5.1 Public Address . . . 415
23.5.2 Port . . . 416
23.5.3 Outbound Ports . . . 416
CONTENTS 15 23.5.5 Join . . . 417 23.5.6 Interfaces . . . 420 23.5.7 SSL . . . 420 23.5.8 Socket Interceptor . . . 420 23.5.9 Symmetric Encryption . . . 420 23.5.10 IPv6 Support . . . 421 23.6 Group Configuration . . . 421 23.7 Map Configuration . . . 422 23.7.1 Map Store . . . 424 23.7.2 Near Cache . . . 424 23.7.3 Indexes . . . 424 23.7.4 Entry Listeners . . . 424 23.8 MultiMap Configuration . . . 424 23.9 Queue Configuration . . . 425 23.10Topic Configuration . . . 426
23.11Reliable Topic Configuration . . . 427
23.12List Configuration . . . 428
23.13Set Configuration . . . 428
23.14Ringbuffer Configuration . . . 429
23.15Semaphore Configuration . . . 430
23.15.1 ReplicatedMap Configuration . . . 430
23.16Executor Service Configuration . . . 431
23.17Cache Configuration . . . 432
23.18Serialization Configuration . . . 433
23.19MapReduce Jobtracker Configuration . . . 434
23.20Services Configuration . . . 435
23.21Management Center Configuration . . . 436
23.22WAN Replication Configuration . . . 436
23.23Enterprise WAN Replication Configuration . . . 437
23.23.1 IMap and ICache WAN Configuration . . . 438
23.24Quorum Configuration . . . 440
23.25Partition Group Configuration . . . 441
23.26Listener Configurations . . . 442
23.27Logging Configuration . . . 446
23.28System Properties . . . 447
16 CONTENTS
24 Extending Hazelcast 451
24.1 User Defined Services . . . 451
24.1.1 Creating the Service Class . . . 451
24.1.2 Enabling the Service Class . . . 452
24.1.3 Adding Properties to the Service . . . 453
24.1.4 Starting the Service . . . 453
24.1.5 Placing a Remote Call via Proxy . . . 453
24.1.6 Creating Containers . . . 458
24.1.7 Partition Migration . . . 462
24.1.8 Creating Backups . . . 466
24.2 WaitNotifyService . . . 469
24.3 Discovery SPI . . . 469
24.3.1 Discovery SPI Interfaces and Classes . . . 469
24.3.2 Discovery Strategy . . . 471
24.3.3 DiscoveryService (Framework integration) . . . 475
24.4 Config Properties SPI . . . 475
24.4.1 Config Properties SPI Classes . . . 475
24.4.2 Config Properties SPI Example . . . 476
25 Network Partitioning - Split Brain Syndrome 479 25.1 Understanding Partition Recreation . . . 479
25.2 Understanding Backup Partition Creation . . . 479
25.3 Understanding The Update Overwrite Scenario . . . 479
25.4 What Happens When The Network Failure Is Fixed . . . 480
25.5 How Hazelcast Split Brain Merge Happens . . . 480
25.6 Specifying Merge Policies . . . 481
26 License Questions 483 26.1 Embedded Dependencies . . . 483
26.2 Runtime Dependencies . . . 483
27 Common Exception Types 485 28 Frequently Asked Questions 487 28.1 Why 271 as the default partition count? . . . 487
28.2 Is Hazelcast thread safe? . . . 487
28.3 What happens when a node goes down? . . . 487
28.4 How do I test the connectivity? . . . 488
28.5 How do I choose keys properly? . . . 488
28.6 How do I reflect value modifications? . . . 488
CONTENTS 17
28.8 Does Hazelcast support hundreds of nodes? . . . 490
28.9 Does Hazelcast support thousands of clients? . . . 490
28.10What is the difference between old LiteMember and new Smart Client? . . . 490
28.11How do you give support? . . . 490
28.12Does Hazelcast persist? . . . 490
28.13Can I use Hazelcast in a single server? . . . 490
28.14How can I monitor Hazelcast? . . . 491
28.15How can I see debug level logs? . . . 491
28.16What is the difference between client-server and embedded topologies? . . . 491
28.17How do I know it is safe to kill the second node? . . . 492
28.18When do I need Native Memory solutions? . . . 492
28.19Is there any disadvantage of using near-cache? . . . 492
28.20Is Hazelcast secure? . . . 492
28.21How can I set socket options? . . . 492
28.22I periodically see client disconnections during idle time? . . . 492
28.23How to get rid of “java.lang.OutOfMemoryError: unable to create new native thread”? . . . 493
28.24Does repartitioning wait for Entry Processor? . . . 493
28.25Why do Hazelcast instances on different machines not see each other? . . . 493
28.26What Does “Replica: 1 has no owner” Mean? . . . 494
Chapter 1
Preface
Welcome to the Hazelcast Reference Manual. This manual includes concepts, instructions and samples to guide you on how to use Hazelcast and build Hazelcast applications.
As the reader of this manual, you must be familiar with the Java programming language and you should have installed your preferred IDE.
1.0.0.0.1 Product Naming Throughout this manual:
• Hazelcastrefers to the open source edition of Hazelcast in-memory data grid middleware. It is also the name of the company providing the Hazelcast product.
• Hazelcast Enterpriserefers to the commercial edition of Hazelcast.
1.0.0.0.2 Licensing Hazelcast is free provided under the Apache 2 license. Hazelcast Enterprise is commercially licensed by Hazelcast, Inc.
For more detailed information on licensing, please see theLicense Questions appendix.
1.0.0.0.3 Trademarks Hazelcast is a registered trademark of Hazelcast, Inc. All other trademarks in this manual are held by their respective owners.
1.0.0.0.4 Customer Support Support for Hazelcast is provided via GitHub, Mail Group and StackOverflow For information on support for Hazelcast Enterprise, please see hazelcast.com.
1.0.0.0.5 Contributing to Hazelcast You can contribute to the Hazelcast code, report a bug or request an enhancement. Please see the following resources.
• Developing with Git: Document that explains the branch mechanism of Hazelcast and how to request changes. • Hazelcast Contributor Agreement form: Form that each contributing developer needs to fill and send back to
Hazelcast.
• Hazelcast on GitHub: Hazelcast repository where the code is developed, issues and pull requests are managed.
1.0.0.0.6 Typographical Conventions Below table shows the conventions used in this manual.
Convention Description
bold font - Indicates part of a sentence that require the reader’s specific attention. - Also indicates property/parameter values.
italic font - When italicized words are enclosed with “<” and “>”, indicates a variable in command or code syntax that you must replace, e.g. hazelcast-<version>.jar. - Note and Related Information texts are in italics.
monospace - Indicates files, folders, class and library names, code snippets, and inline code words in a sentence. 19
20 CHAPTER 1. PREFACE
Convention Description
RELATED INFORMATION - Indicates a resource that is relevant to the topic, usually with a link or cross-reference.
NOTE Indicates information that is of special interest or importance, e.g. an additional action required only in certain circumstances.
Chapter 2
What’s New in Hazelcast 3.6
This chapter includes the release notes, information on how to upgrade Hazelcast from previous releases and the revision history for this document.
2.1
Release Notes
This section lists the new features and enhancements developed and bugs fixed for this release.
2.1.1
New Features
The following the new features introduced with Hazelcast 3.6 release.
• High-Density Memory Store for Hazelcast Map: With this release, Hazelcast Map data structure is now equipped with the High-Density Memory Store, previously implemented for Hazelcast JCache. Please refer to theSetting In Memory Format section.
• Discovery Service Provider Interface (Discovery SPI): You can use this SPI to discover Hazelcast instances on cloud environments provided by jclouds®, Kubernetes and many more. The existing discovery mechanisms that Hazelcast provides (Multicast, TCP/IP and Amazon EC2) have been re-implemented on top of this new Discovery SPI. Please refer to theDiscovery SPI section.
• Client Protocol: This feature presents the Hazelcast’s new open binary client protocol. Please refer to Open Binary Client Protocol Documentation.
• Client Cross Version Compatibility: Now you can upgrade your Hazelcast clients independently from servers and other clients. Please refer to Open Binary Client Protocol Documentation.
• Support for cloud providers through jclouds®: Hazelcast now supports deployments on all the well-known cloud providers through the jclouds® open source library. Please refer to theDiscovering Members with jclouds section.
• Hot Restart Store: This new feature provides fast restarting of the Hazelcast clusters. This is achieved by storing the state of the cluster members to the disk. Please refer to theHot Restart Store section for more details.
• Ringbuffer and Reliable Topic in Hazelcast Clients: The data structures Ringbuffer and Reliable Topic recently introduced by Hazelcast (with the release 3.5) are now implemented for Hazelcast Java Client. Ringbuffer has also been implemented for .NET Client.
• Cluster Quorum for Hazelcast JCache: Cluster Quorum checks are now provided for Hazelcast JCache implementations, too. Please refer to theDefining a Cluster Quorum sectionto refresh and to theICache Configuration sectionto learn configuring it for JCache.
• Split Brain Syndrome handler for Hazelcast JCache: Now Split Brain Syndrome is handled in JCache as it is taken care in Hazelcast Map. Please refer to theJCache Split-Brain section.
• Partition Lost Listener for Hazelcast JCache: You can listen to partition lost events fired in your Hazelcast JCache implementation. Please refer to theICache Configuration section.
22 CHAPTER 2. WHAT’S NEW IN HAZELCAST 3.6 • Hazelcast Docker image: Now you can run Hazelcast using our image in the Docker platform. Please refer
to here.
• Lite Members: With the introduction of Hazelcast Lite Members, you are able to specify certain members in your cluster so that they do not store data. You can use these lite members mostly for your task executions and listener registrations. Please refer toEnabling Lite Members.
• Querying in collections and arrays: Hazelcast is now able to query and index attributes of objects stored in a collection or array. Please refer to theQuerying in collections section.
• Custom attributes extraction: It is now possible to extract a value of an object’s attribute using a custom extractor class. Please refer to theCustom attributes.
• Acquiring locks with a lease time: Now, you can try to acquire locks with a lease time. Please refer to the the comment for the methodtryLock()in ILock code.
• Monitoring the WAN replication: You can now monitor the state of your WAN replications using the Hazelcast Management Center. Please refer to theMonitoring WAN Replication section.
2.1.2
Enhancements
The following are the the enhancements performed for Hazelcast 3.6 release.
• Replicated Map improvements: The implementation of Hazelcast replicated maps has been revisited. Please especially refer to theConsiderations for Replicated Map section.
• Management Center improvements: Alerting mechanism added. Please refer to theManagement Center section.
• Paging Predicate improvements: With the performed improvements, now random page accessing is supported. Please refer to theFiltering with Paging Predicates section.
• Rule based query optimizations: This improvement introduces a query optimizer based on static rewriting rules. The optimizer treats predicates as immutable and returns a modified copy when the optimized one is found. Please refer to thehazelcast.query.optimizer.typeproperty definition in the System Properties section.
• WAN replication improvements: With the improvements performed on Hazelcast’s WAN replication feature, you can now monitor WAN replication events for each data structure and WAN replication now supports different acknowledge types for each target cluster group. Please refer to theWAN Replication Event Filtering API sectionandWAN Replication Acknowledge Types sectionfor more information. • Improvements on Hazelcast’s OSGI support: With this improvement, Hazelcast bundles provide OSGI
services so that the users can manage (create, access, shutdown) the Hazelcast instances through this service on OSGI environments. Having thehazelcast.osgi.startproperty enabled, when an Hazelcast OSGI service is activated, a default Hazelcast instance is created automatically. These instances can be served as an OSGI service to be accessed by other bundles. Registering the created Hazelcast instances behavior is enabled by default and can be disabled using thehazelcast.osgi.register.disabledproperty. Each Hazelcast bundle provides a different OSGI service and their instances can be grouped (clustered) together to prevent possible compatibility issues between different Hazelcast versions/bundles. This grouping behavior is enabled by default and can be disabled using the hazelcast.osgi.grouping.disabled property. Hazelcast OSGI service’s lifecycle (and also the owned/created instances’ lifecycles) are the same as the owner Hazelcast bundles. When the bundle is stopped (deactivated), owned service and Hazelcast instances are also deactivated/shutdown and deregistered automatically. Then, when the bundle is re-activated, its service is registered again. In addition, the Hazelcast Enterprise JAR file is also an OSGI bundle like the Hazelcast OSS JAR file. The following are the other improvements performed to solve the enhancement issues opened by the Hazelcast customers/team.
• There exists a misleading log entry for the methodEventServiceImpl.sendEvent: “logFailure(”IO Queue overloaded! Failed to send event packet to: %s“, subscriber);”. However, the failure is not about I/O queue, it is about connection drop. [6723]
• Approximatemax-sizecalculation should be removed for IMap eviction. [6463]
• SpringAwareWebFiltershould have a constructor which takes properties as arguments. [6438]
• Client side and server side cache proxies handleputAlloperation one by one. This is not efficient. Records for this operation should be grouped as per their partitions and should be sent and processed in batches. [6367]
2.1. RELEASE NOTES 23 • Not requested events should not be sent toMapListener[6349]
• Inconsistent and potentially buggy design inBasicCompletableFuture. [6080]
• Starting with “hazelcast-wm 3.3”, OSGI Manifest Spring package imports should be optional. [6072] • The new client determines the partition ID for every invocation for data structures like queue and list where
the partition ID is static. There is no need for this behavior. It should calculate the partition ID for once when the proxy is created and continue to re-use it. [5848]
• Map.Entrysupplied to Entry Processor is not Serializable any more. [5611]
• The configuration fileminimal-jsonwith the provided scope is not picked up by the shadeplugin. [5543]
• In Spring configuration, when a boolean property is injected forhazelcastbean (<hz:hazelcast:....</hz:hazelcast)
a SAXParseexception is thrown. [5528]
• Currently, key/value pairs are deserialized prior to the execution of entry processor by default. This leads to the need of domain object at the server side, even if entry processor never uses it. [5301]
• In Spring XML configuration, the attributes of socket-optionsshould be of typexs:string. [4700] • ClientMembershipEventdoes not need to have thememberfield. [4282]
• Hazelcast haslockwith lease time feature but does not supporttryLockwith lease time. [1564]
2.1.3
Fixes
The following are the issues solved for Hazelcast 3.6 release. 3.6-EA2 Fixes
• MapLoader may insert null values into IMap causing memory leak. [6830]
• When replicated map entries are migrated to a new destination; TTL eviction should be scheduled, eviction should be retried when a failure caused by the migration happens and the sync interval should be increased. [6799]
• There is a logical error in the methodRingbuffer.readManyAsync()when minSize = 0. In this case, the Ringbuffer is not read and nothing is returned. [6787]
• When a listener’s registration is made from the listener configuration, an error occurs during the listener initialization. [6784]
• Remaining cache invalidation messages should be flushed on theICacheServicewhile the member is in the SHUTTING_DOWNstate. [6778]
• When a client cannot send a request to one of the connections,TargetNotMemberException is thrown. This name is confusing the Hazelcast users. [6766]
• ClassCastExceptionis thrown when using TimestampwithinDataSerializable. [6759]
• The methoddestroyDistributedObject()of ReplicatedMapServiceiterates over partition containers and record stores and destroys them. While destroying, record store callsdestroyDistributedObject()which leads to an infinite loop. [6754]
• Hazelcast does not inject its instance intoHazelcastInstanceAwareregistered via classname. [6697] • There is a sporadic startup failure in 3.6-EA. [6684]
• There is no need to useCacheLoaderinside the client/server side cache proxies. [6676]
• Fixed wrong calculation of eviction removal size whenPER_NODE max-sizepolicy is used. [6675]
• If the cluster state is not activeRepartitioningTask should not be triggered. Otherwise, it causes infinite retries and prevents the member from shtutdown. [6663]
• There are broken XML configuration tests in the Hazelcast client package. [6633]
• There is a memory leak since the methodpublishBathcedEventsdoes not remove the events frombatchEvent. [6618]
• Custom credentials class is not de-serialized on the server side. [6615]
• Lite member element should be added to the Hazelcast Spring configuration. [6605]
• EntryListenershows the unprocessed value in combination withPostProcessingMapStore. [6588] • Clients cannot submitHazelcastInstanceAwarecallables. [6570]
3.6-EA1 Fixes
24 CHAPTER 2. WHAT’S NEW IN HAZELCAST 3.6 • HazelcastCachingProviderdoes not use the specified instance (by the object) when instance-nameis not
specified. [6454]
• onExecutionFailureshould be called before returning fromrun, if backup is not valid. [6420]
• OperationThread.priorityPendingCount() should return scheduleQueue.prioritySize() instead of scheduleQueue.normalSize(). [6318]
• There is a growth in heap usage caused by a memory leak in the following scenario: A node in the cluster regularly creates maps and puts entries into it, again in regular intervals. Another node removes the entries minutes after they were put, and if the map is empty, it destroys the map. [6317]
• Currently, there is anEntryEvictedListenerthat is notified both for expiration and eviction events. There should be a separate listener for expired entries: eviction happens due to size constraints, and expiry is once the entry has expired. [6311]
• InvocationFutures async calls do not detect the lost operations. [6250]
• When the method setBooleanAttribute of the class Member is run, Null Pointer Exception is occurred on STDOUT. The problem is in the methodsendMemberAttributeEvent of the class ClusterServiceImpl. [6223]
• IOBalancerkeeps references of all the socket reader/writers but when destroying the connection, they release the references for only the ones which has endpoints. This causes a memory leak. [6199]
• ILIKEandRegexexamples should be added to the Reference Manual under the “Supported SQL Syntax” section. [6190]
• GroupPropertydefaulting does not work properly when programmatic configuration is used. [6174]
• When integrating Hazelcast in Spring Boot: if HazelcastInstance is created using the default newHazelcastInstance static method, then an HazelcastInstance whose Config has a valid configurationUrl property is created. However, XmlBuilder does not set this URL in the configu-ration it parses. [6061]
• Hazelcast’s latest snapshot run fails due to the introduction of ClientExceptionFactorywhich has been developed for exception processing and working well in that sense. [6010]
• The classHazelcastXATesthas only fast and slow modes (nothing in between) and possibly due to this, sometimes a transaction is waiting for a timeout. Either the transaction recovery or the test class itself is racy. [5923]
• A memory leak occurs when a listener is added and removed from client. A “remove” runnable in the collection that is stored inClientEndpointImplis the leftover. This runnable collection is used to cleanup the listeners when client is disconnected, it should be removed too after the listener is removed. [5893]
• The class CacheRemoveAllOperation does not send the “completed” event in some cases, e.g. if CacheRecordStorefor that partition is not created yet or if the filtered keys are empty. [5865]
• In the class MapProxyImpl, the methods executeOnKey and submitToKey create an EntryOperation with the thread ID set. This does not happen with the class ClientMapProxy. Therefore, the class MapExecuteOnKeyRequestshould take a thread ID and set this on the generatedEntryOperation. [5857] • The methodIndexImpl.getRecords()fails with Null Pointer Exception due to the inconsistency between
thenot(...equals())andnotEquals(). [5807]
• The methodHazelcastHttpSession.getAttribute()for WebFilter does not work whendeferredWriteis set totrue. [5798]
• When hazelcast.nio.faststring is enabled, UTFEncoderDecoder tries to create a FastStringCreator. However, if the reflection is not available due to the security manager,buildFastStringCreatorreturns null and consequentlyStringCreatorbecomes null. [5777]
• hazelcast-jca-rar/pom.xmlreferences tosrc/main/rar/ra.xmlwhich does not exist. [5760]
• The Maven profile mvn clean compile -Pqa does not exist but it is documented in the README of Hazelcast. [5746]
• PerformanceLogFileonly compiles if JDK 1.7 or above is used. [5729]
• Currently, for every deserialization aBufferObjectDataInputis created. This generates waste since it is created with an array of data for every deserialization. TheBufferObjectDataOutputis already cached; the input should use a similar approach. [5562]
• When any entities are defined as read only in the Hibernate L2 cache, an invalidation of the cache (such as caused by executing a native SQLQuery) leads to the errorUnsupportedOperationException. [5562] • The performance impacts of TWO_PHASE and LOCAL transaction types should be documented. [5075] • Client requests are very inefficient when determining the partition ID. [4940]
-2.2. UPGRADING HAZELCAST 25 it always returns both keys and values. This can lead to unnecessary load and potentially even an OOM exception. [4642]
• Hazelcast is stuck in TIMED_WAITING when used as 2nd level cache for Hibernate. [4406] • Management Center license loading problem when REST API is used. [189]
• Executor monitoring in Management Center does not show the “cancelled” operations" [177]
• When an alert for a data structure (map, queue, etc.) with its specific name is created, a NullPointerExceptionis thrown after the cluster is reset. [175]
• Default directory name is hardcoded as “mancenter3.5” and it needs to be maintained for every major release. This process should be dynamic. [174]
• Throughput statistics for Map shows nothing when theputAll()method is used. [159]
2.2
Upgrading Hazelcast
In the following sections, you can see the changes that you should take into account before upgrading to latest Hazelcast from 2.x and 3.x releases.
2.2.1
Upgrading from 2.x
• Removal of deprecated static methods: The static methods of Hazelcast class reaching Hazelcast data components have been removed. The functionality of these methods can be reached from HazelcastInstance interface. Namely you should replace following:
Map<Integer, String> customers = Hazelcast.getMap( "customers" ); with
HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(); // or if you already started an instance named "instance1"
// HazelcastInstance hazelcastInstance = Hazelcast.getHazelcastInstanceByName( "instance1" ); Map<Integer, String> customers = hazelcastInstance.getMap( "customers" );
• Removal of lite members: With 3.0 there will be no member type as lite member. As 3.0 clients are smart client that they know in which node the data is located, you can replace your lite members with native clients. • Renaming “instance” to “distributed object”: Before 3.0 there was a confusion for the term “instance”. It was used for both the cluster members and the distributed objects (map, queue, topic, etc. instances). Starting 3.0, the term instance will be only used for Hazelcast instances, namely cluster members. We will use the term “distributed object” for map, queue, etc. instances. So you should replace the related methods with the new renamed ones. As 3.0 clients are smart client that they know in which node the data is located, you can replace your lite members with native clients.
public static void main( String[] args ) throws InterruptedException { HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(); IMap map = hazelcastInstance.getMap( "test" );
Collection<Instance> instances = hazelcastInstance.getInstances(); for ( Instance instance : instances ) {
if ( instance.getInstanceType() == Instance.InstanceType.MAP ) {
System.out.println( "There is a map with name: " + instance.getId() ); }
} } with
26 CHAPTER 2. WHAT’S NEW IN HAZELCAST 3.6
public static void main( String[] args ) throws InterruptedException { HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(); IMap map = hz.getMap( "test" );
Collection<DistributedObject> objects = hazelcastInstance.getDistributedObjects(); for ( DistributedObject distributedObject : objects ) {
if ( distributedObject instanceof IMap ) {
System.out.println( "There is a map with name: " + distributedObject.getName() ); }
} }
• Package structure change: PartitionService has been moved to package com.hazelcast.core from com.hazelcast.partition.
• Listener API change: Before 3.0,removeListenermethods was taking the Listener object as parameter. But, it causes confusion as same listener object may be used as parameter for different listener registrations. So we have changed the listener API.addListenermethods return you an unique ID and you can remove listener by using this ID. So you should do following replacement if needed:
IMap map = hazelcastInstance.getMap( "map" ); map.addEntryListener( listener, true ); map.removeEntryListener( listener ); with
IMap map = hazelcastInstance.getMap( "map" );
String listenerId = map.addEntryListener( listener, true ); map.removeEntryListener( listenerId );
• IMap changes:
• tryRemove(K key, long timeout, TimeUnit timeunit)returns boolean indicating whether operation is successful.
• tryLockAndGet(K key, long time, TimeUnit timeunit)is removed. • putAndUnlock(K key, V value)is removed.
• lockMap(long time, TimeUnit timeunit)andunlockMap() are removed.
• getMapEntry(K key)is renamed asgetEntryView(K key). The returned object’s type, MapEntry class is renamed as EntryView.
• There is no predefined names for merge policies. You just give the full class name of the merge policy implementation.
<merge-policy>com.hazelcast.map.merge.PassThroughMergePolicy</merge-policy>
Also MergePolicy interface has been renamed to MapMergePolicy and also returning null from the implemented merge()method causes the existing entry to be removed.
• IQueue changes: There is no change on IQueue API but there are changes on howIQueueis configured. With Hazelcast 3.0 there will not be backing map configuration for queue. Settings like backup count will be directly configured on queue config. For queue configuration details, please see theQueue section.
• Transaction API change: In Hazelcast 3.0, transaction API is completely different. Please see the Transactions chapter.
• ExecutorService API change: Classes MultiTask and DistributedTask have been removed. All the functionality is supported by the newly presented interface IExecutorService. Please see theExecutor Service section.
• LifeCycleService API:The lifecycle has been simplified. pause(),resume(), restart()methods have been removed.
2.3. DOCUMENT REVISION HISTORY 27 • ICountDownLatch: await()operation has been removed. We expect users to useawait()method with
timeout parameters.
• ISemaphore API: TheISemaphore has been substantially changed. attach(),detach()methods have been removed.
• In 2.x releases, the default value formax-sizeeviction policy wascluster_wide_map_size. In 3.x releases, default isPER_NODE. After upgrading, themax-sizeshould be set according to this new default, if it is not changed. Otherwise, it is likely that OutOfMemory exception may be thrown.
2.2.2
Upgrading from 3.x
• Introducing the spring-aware element: Before the release 3.5, Hazelcast usesSpringManagedContext
to scanSpringAware annotations by default. This may cause some performance overhead for the users who do not useSpringAware. This behavior has been changed with the release of Hazelcast 3.5. SpringAware annotations are disabled by default. By introducing thespring-awareelement, now it is possible to enable it by adding the<hz:spring-aware />tag to the configuration. Please see theSpring Integration section.
2.3
Document Revision History
Chapter Section Description
Chapter 3 - Getting Started Phoning Home Added as a new section to explain phone home data.
Deploying using Docker Added as a new section to describe how you can deploy your Hazelcast projects using the Docker platform. Chapter 5 - Hazelcast Clusters Discovering Members with jclouds Added as a new section to explain how you can enable Hazelcast to discover your members with jclouds®.
Chapter 6 - Distributed Data Structures Map Map Eviction section updated by adding the definition for themin-eviction-check-millisproperty. The sectionUnderstanding Map Evictionadded for a more clearer explanation of the eviction mechanism. Setting In Memory Format sectionupdated by adding the information on how to configure Hazelcast Map to use High-Density Memory Store. Creating Near Cache for Map updated by adding a note related to enabling near caches on a lite member.
Lock Added the explanation for the methodtryLockwith lease.
Replicated Map Replicating instead of Partitioningupdated by adding a note related to replicated map usage in a lite member. Chapter 8 - Distributed Computing Selecting Members for Task Execution Added a paragraph on how to select a lite member.
Chapter 9 - Distributed Query Filtering with Paging Predicates The note stating that the random page accessing is not supported removed, since it is now supported with this release. Chapter 10 - Transactions ONE_PHASE vs. TWO_PHASE Added as a new section explaining the trade offs between these two transaction types.
Creating a Transaction Interface Replaced the transaction type name LOCAL with ONE_PHASE. Updated the definitions of transaction types. Chapter 11 - Hazelcast JCache ICache Partition Lost Listener Added as a new section explaining how to listen when a partition is lost in a Hazelcast JCache implementation.
JCache Split-Brain Added as a new section.
Chapter 12 - Integrated Clustering Web Session Replication Marking Transient Attributesadded as a new section.
Spring Integration Declarative Hazelcast JCache Based Caching Configurationadded as a new section.
Hibernate Second Level Cache Added additional information related to Hibernate Native Client mode to the introduction paragraph of the Setting Client/Server for Hibernate section. Chapter 13 - Storage Hot Restart Store Added as a new section to explain how Hazelcast’s Hot Restart feature works and its design details.
Chapter 14 - Hazelcast Java Client Hazelcast Clients Feature Comparison Added as a new section.
Client Network Configuration Updated by adding the definition of the new IAM role configuration element. Chapter 15 - Other Client Implementations Windows C++ Client Updated by adding static/dynamic library related flag information.
Chapter 17 - Management Defining a Cluster Quorum Added information on quorum support for caches. Added the definition of the new configuration elementquorum-reftoICache Configuration section.
Management Center A note on how to see the cache statistics on the Management Center added under theCaches section.Replicated Maps added as a new section. Added the information explaining the new e-mail notification mechanism toCreating Alerts.Monitoring WAN Replicationadded as a new section.Hot Restartadded as a new section. Monitoring with JMX MBean Naming for Hazelcast Data StructuresandConnecting to JMX Agentadded as new sections.
Enabling Lite Members Added as a new section. AlsoData PartitioningandPartition Tablesections updated to include Lite Member related information.
Chapter 21 - WAN WAN Replication Event Filtering API sectionandWAN Replication Acknowledge Types sectionadded as new sections to explain the improvements performed for Hazelcast WAN replication feature. A note related to the methodsclear,destroyandevictAlladded to theWAN Replication Additional Information section.
Chapter 22 - OSGI Added as a new chapter.
Chapter 23 - Hazelcast Configuration Improved by adding missing configuration elements and attributes. AddedQuorum Configurationsection. Network Configuration Joinsection updated by adding the explanation fordiscovery-strategieselement.
System Properties Added the new system propertieshazelcast.unsafe.mode,hazelcast.io.input.thread.count,hazelcast.io.output.thread.count andhazelcast.query.optimizer.type. Updated the definition of hazelcast.io.thread.count. Chapter 24 - Extending Hazelcast This title added as a chapter to include the section previously present as “User Defined Services”.
Discovery SPI Added as a new section. Config Properties SPI Added as a new section.
Chapter 28 - FAQ Added new questions/answers.
Chapter 3
Getting Started
This chapter explains how to install Hazelcast, start a Hazelcast member and client, and gives Hazelcast configuration fundamentals.
3.1
Installation
The following sections explains the installation of Hazelcast and Hazelcast Enterprise.
3.1.1
Hazelcast
You can find Hazelcast in standard Maven repositories. If your project uses Maven, you do not need to add additional repositories to yourpom.xmlor addhazelcast-<version>.jarfile into your classpath (Maven does that
for you). Just add the following lines to yourpom.xml:
<dependencies> <dependency>
<groupId>com.hazelcast</groupId> <artifactId>hazelcast</artifactId> <version>3.6</version>
</dependency> </dependencies>
As an alternative, you can download and install Hazelcast yourself. You only need to:
• Download `hazelcast-<`*version*`>.zip` file from <a href="www.hazelcast.org" target="_blank">hazelcast.org</a>. • Unzip `hazelcast-<`*version*`>.zip` file.
• Add `hazelcast-<`*version*`>.jar` file into your classpath.
3.1.2
Hazelcast Enterprise
There are two Maven repositories defined for Hazelcast Enterprise: <repository>
<id>Hazelcast Private Snapshot Repository</id>
<url>https://repository-hazelcast-l337.forge.cloudbees.com/snapshot/</url> </repository>
<repository>
30 CHAPTER 3. GETTING STARTED <id>Hazelcast Private Release Repository</id>
<url>https://repository-hazelcast-l337.forge.cloudbees.com/release/</url> </repository>
Hazelcast Enterprise customers may also define dependencies, a sample of which is shown below.
<dependency> <groupId>com.hazelcast</groupId> <artifactId>hazelcast-enterprise-tomcat6</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>com.hazelcast</groupId> <artifactId>hazelcast-enterprise-tomcat7</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>com.hazelcast</groupId> <artifactId>hazelcast-enterprise</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>com.hazelcast</groupId> <artifactId>hazelcast-enterprise-all</artifactId> <version>${project.version}</version> </dependency>
3.1.2.1 Setting the License Key
To use Hazelcast Enterprise, you need to set the license key in configuration.
• **Declarative Configuration**
<hazelcast> ...
<license-key>HAZELCAST_ENTERPRISE_LICENSE_KEY</license-key> ...
</hazelcast>
• **Client Declarative Configuration**
<hazelcast-client> ...
<license-key>HAZELCAST_ENTERPRISE_LICENSE_KEY</license-key> ...
</hazelcast-client>
• **Programmatic Configuration**
Config config = new Config();
config.setLicenseKey( "HAZELCAST_ENTERPRISE_LICENSE_KEY" ); • **Spring XML Configuration**
3.2. STARTING THE MEMBER AND CLIENT 31 <hz:config>
...
<hz:license-key>HAZELCAST_ENTERPRISE_LICENSE_KEY</hz:license-key> ...
</hz:config>
• **JVM System Property**
-Dhazelcast.enterprise.license.key=HAZELCAST_ENTERPRISE_LICENSE_KEY
3.2
Starting the Member and Client
Having installed Hazelcast, you can get started.
In this short tutorial, you perform the following activities.
1. Create a simple Java application using the Hazelcast distributed map and queue. 2. Run our application twice to have a cluster with two members (JVMs).
3. Connect to our cluster from another Java application by using the Hazelcast Native Java Client API.
Let’s begin.
• The following code starts the first Hazelcast member and creates and uses thecustomersmap and queue.
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance; import java.util.Map;
import java.util.Queue; public class GettingStarted {
public static void main( String[] args ) {
HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(); Map<Integer, String> customers = hazelcastInstance.getMap( "customers" ); customers.put( 1, "Joe" );
customers.put( 2, "Ali" ); customers.put( 3, "Avi" );
System.out.println( "Customer with key 1: " + customers.get(1) ); System.out.println( "Map Size:" + customers.size() );
Queue<String> queueCustomers = hazelcastInstance.getQueue( "customers" ); queueCustomers.offer( "Tom" );
queueCustomers.offer( "Mary" ); queueCustomers.offer( "Jane" );
System.out.println( "First customer: " + queueCustomers.poll() ); System.out.println( "Second customer: "+ queueCustomers.peek() ); System.out.println( "Queue size: " + queueCustomers.size() ); }
}
• Run thisGettingStartedclass a second time to get the second member started. The members form a cluster and the output is similar to the following.
32 CHAPTER 3. GETTING STARTED Members [2] {
Member [127.0.0.1:5701] Member [127.0.0.1:5702] this }
• Now, add the hazelcast-client-<version>.jar library to your classpath. This is required to use a Hazelcast client.
• The following code starts a Hazelcast Client, connects to our cluster, and prints the size of thecustomers map. package com.hazelcast.test; import com.hazelcast.client.config.ClientConfig; import com.hazelcast.client.HazelcastClient; import com.hazelcast.core.HazelcastInstance; import com.hazelcast.core.IMap;
public class GettingStartedClient {
public static void main( String[] args ) {
ClientConfig clientConfig = new ClientConfig();
HazelcastInstance client = HazelcastClient.newHazelcastClient( clientConfig ); IMap map = client.getMap( "customers" );
System.out.println( "Map Size:" + map.size() ); }
}
• When you run it, you see the client properly connecting to the cluster and printing the map size as3. Hazelcast also offers a tool,Management Center, that enables you to monitor your cluster. To use it, deploy themancenter-<version>.warincluded in the ZIP file to your web server. You can use it to monitor your maps, queues, and other distributed data structures and members. Please see theManagement Center section for usage explanations.
By default, Hazelcast uses Multicast to discover other members that can form a cluster. If you are working with other Hazelcast developers on the same network, you may find yourself joining their clusters under the default settings. Hazelcast provides a way to segregate clusters within the same network when using Multicast. Please see theCreating Cluster Groups for more information. Alternatively, if you do not wish to use the default Multicast mechanism, you can provide a fixed list of IP addresses that are allowed to join. Please see theJoin Configuration sectionfor more information.
RELATED INFORMATION
You can also check the video tutorials here.
3.2.1
Deploying On Amazon EC2
You can deploy your Hazelcast project onto Amazon EC2 environment using Third Party tools such as Vagrant and Chef.
You can find a sample deployment project (amazon-ec2-vagrant-chef) with step by step instructions in the hazelcast-integrationfolder of the hazelcast-code-samples package. Please refer to this sample project for more information.
3.3
Configuring Hazelcast
3.4. PHONING HOME 33 • First, it looks for thehazelcast.configsystem property. If it is set, its value is used as the path. This is useful if you want to be able to change your Hazelcast configuration: you can do this because it is not embedded within the application. You can set theconfigoption with the following command:
- Dhazelcast.config=<path to the hazelcast.xml>.
The path can be a normal one or a classpath reference with the prefix classpath:.
• If the above system property is not set, Hazelcast then checks whether there is ahazelcast.xmlfile in the working directory.
• If not, then it checks whetherhazelcast.xmlexists on the classpath.
• If none of the above works, Hazelcast loads the default configuration, i.e. hazelcast-default.xml that comes withhazelcast.jar.
When you download and unziphazelcast-<version>.zip, you will see ahazelcast.xmlin the/binfolder. This
is the declarative configuration file for Hazelcast. Part of this XML file is shown below.
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.5.xsd"
xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <group>
<name>dev</name>
<password>dev-pass</password> </group>
<management-center enabled="false">http://localhost:8080/mancenter</management-center> <network>
<port auto-increment="true" port-count="100">5701</port> <outbound-ports>
<!--Allowed port range when connecting to other nodes. 0 or * means use system provided port.
-->
<ports>0</ports> </outbound-ports> <join>
<multicast enabled="true">
<multicast-group>224.2.2.3</multicast-group> <multicast-port>54327</multicast-port> </multicast>
<tcp-ip enabled="false">
For most users, default configuration should be fine. If not, you can tailor this XML file according to your needs by adding/removing/modifying properties.
Besides declarative configuration, you can configure your cluster programmatically. Just instantiate aConfigobject and add/remove/modify properties.
You can also use wildcards while configuring Hazelcast. Please refer to theUsing Wildcard sectionfor details. Hazelcast also offers System Properties to tune some aspects of it. Please refer to theSystem Properties sectionfor details.
RELATED INFORMATION
Please refer to theHazelcast Configuration chapterfor more information.
3.4
Phoning Home
34 CHAPTER 3. GETTING STARTED Hazelcast member instances call our phone home server initially when they are started and then for every 24 hours. This applies to all the instances joined to the cluster.
What is sent in?
The following information is sent in a phone home: • Hazelcast version
• Local Hazelcast member UUID • Download ID
• A hash value of the cluster ID
• Cluster size bands for 5, 10, 20, 40, 60, 100, 150, 300, 600 and > 600
• Number of connected clients bands of 5, 10, 20, 40, 60, 100, 150, 300, 600 and > 600 • Cluster uptime
• Member uptime
– Hazelcast Enterprise specific:
– Number of clients by language (Java, C++, C#) – Flag for Hazelcast Enterprise
– Hash value of license key – Native memory usage Phone Home Code
The phone home code itself is open source. Please see here. Disabling Phone Homes
Set thehazelcast.phone.home.enabledsystem property to false either in the config or on the Java command line. Please see theSystem Properties section for information on how to set a property.
Phone Home URLs
For versions 1.x and 2.x: http://www.hazelcast.com/version.jsp.
For versions 3.x up to 3.6: http://versioncheck.hazelcast.com/version.jsp. For versions after 3.6: http://phonehome.hazelcast.com/ping.
Chapter 4
Hazelcast Overview
Hazelcast is an open source In-Memory Data Grid (IMDG). It provides elastically scalable distributed In-Memory computing, widely recognized as the fastest and most scalable approach to application performance. Hazelcast does this in open source. More importantly, Hazelcast makes distributed computing simple by offering distributed implementations of many developer friendly interfaces from Java such as Map, Queue, ExecutorService, Lock, and JCache. For example, the Map interface provides an In-Memory Key Value store which confers many of the advantages of NoSQL in terms of developer friendliness and developer productivity.
In addition to distributing data In-Memory, Hazelcast provides a convenient set of APIs to access the CPUs in your cluster for maximum processing speed. Hazelcast is designed to be lightweight and easy to use. Since Hazelcast is delivered as a compact library (JAR) and since it has no external dependencies other than Java, it easily plugs into your software solution and provides distributed data structures and distributed computing utilities.
Hazelcast is highly scalable and available (100% operational, never failing). Distributed applications can use Hazelcast for distributed caching, synchronization, clustering, processing, pub/sub messaging, etc. Hazelcast is implemented in Java and has clients for Java, C/C++, .NET and REST. Hazelcast also speaks memcache protocol. It plugs into Hibernate and can easily be used with any existing database system.
If you are looking for In-Memory speed, elastic scalability, and the developer friendliness of NoSQL, Hazelcast is a great choice.
Hazelcast is simple
Hazelcast is written in Java with no other dependencies. It exposes the same API from the familiar Java util package, exposing the same interfaces. Just add hazelcast.jarto your classpath, and you can quickly enjoy JVMs clustering and you can start building scalable applications.
Hazelcast is Peer-to-Peer
Unlike many NoSQL solutions, Hazelcast is peer-to-peer. There is no master and slave; there is no single point of failure. All nodes store equal amounts of data and do equal amounts of processing. You can embed Hazelcast in your existing application or use it in client and server mode where your application is a client to Hazelcast nodes. Hazelcast is scalable
Hazelcast is designed to scale up to hundreds and thousands of nodes. Simply add new nodes and they will automatically discover the cluster and will linearly increase both memory and processing capacity. The nodes maintain a TCP connection between each other and all communication is performed through this layer.
Hazelcast is fast
Hazelcast stores everything in-memory. It is designed to perform very fast reads and updates. Hazelcast is redundant
Hazelcast keeps the backup of each data entry on multiple nodes. On a node failure, the data is restored from the backup and the cluster will continue to operate without downtime.
36 CHAPTER 4. HAZELCAST OVERVIEW
4.1
Sharding in Hazelcast
Hazelcast shards are called Partitions. By default, Hazelcast has 271 partitions. Given a key, we serialize, hash and mode it with the number of partitions to find the partition which the key belongs to. The partitions themselves are distributed equally among the members of the cluster. Hazelcast also creates the backups of partitions and distributes them among nodes for redundancy.
RELATED INFORMATION
Please refer to theData Partitioning sectionfor more information on how Hazelcast partitions your data.
4.2
Hazelcast Topology
You can deploy a Hazelcast cluster in two ways: Embedded or Client/Server.
If you have an application whose main focal point is asynchronous or high performance computing and lots of task executions, then Embedded deployment is useful. In this type, members include both the application and Hazelcast data and services. The advantage of the Embedded deployment is having a low-latency data access.
See the below illustration.
Figure 4.1:
In the Client/Server deployment, Hazelcast data and services are centralized in one or more server members and they are accessed by the application through clients. You can have a cluster of server members that can be independently created and scaled. Your clients communicate with these members to reach to Hazelcast data and services on them. Hazelcast provides native clients (Java, .NET and C++), Memcache clients and REST clients. See the below illustration.
Client/Server deployment has advantages including more predictable and reliable Hazelcast performance, easier identification of problem causes, and most importantly, better scalability. When you need to scale in this deployment type, just add more Hazelcast server members. You can address client and server scalability concerns separately. If you want low-latency data access, as it is in the Embedded deployment, and you also want the scalability advantages of the Client/Server deployment, you can consider to define near caches for your clients. This enables the frequently used data to be kept in the client’s local memory. Please refer toConfiguring Client Near Cache.
