Hazelcast releases IMDG 3.10 – introducing new data structures, flake ID generator and CRDT, as well as enhanced split-brain protection
Greg Luck of Hazelcast
Hazelcast, the open source in-memory data management company with tens of thousands of installed clusters and over 55 million server starts per month, announced that Hazelcast IMDG 3.10 is generally available as a production ready build.Major themes, which have been driven by open source community input, include the release of a conflict-free replicated data type (CRDT), a Flake ID Generator and several split-brain protection enhancements.
IMDG 3.10 is the first version of Hazelcast IMDG to include a CRDT implementation – CRDT PN-Counter (positive-negative counter). CRDT is a specially-designed data structure used to achieve strong eventual consistency. With this new data structure, updates are local on a Hazelcast member.
The CRDT PN-Counter allows counting (increment/decrement/query) operations and guarantees that, once all members are connected, the value on all counters is the same. Since operations are local, the CRDT PN-Counter provides consistent low latency and high throughput enabling applications to achieve high performance regardless of load (such as counting likes, page views or connected user count). Additional CRDT data structures will follow in future releases.
IMDG 3.10 also introduces the Hazelcast Flake ID Generator, a tool used to generate cluster-wide unique identifiers. In single-node systems, unique ID generation is a relatively straightforward process. However, problems can occur in distributed implementations, for example preserving uniqueness, eliminating contention, and maintaining safety during failures. The Hazelcast Flake ID Generator replaces the existing ID Generator. It generates IDs without coordination between Hazelcast members, meaning it can generate unique IDs even during split-brain and network partitioning failures.
In response to community feedback about split-brain protection and overall split-brain functionality, Hazelcast has introduced several new split-brain enhancements in its latest release. Previously only Map, ICache and ReplicatedMap supported merging after a network partition had occurred. In IMDG 3.10, split-brain merging policies are available for a host of new structures: ISet, IList, IQueue, Ringbuffer, MultiMap, IAtomicLong, IAtomicReference, IScheduledExecutorService and CardinalityEstimator.
In addition to providing out-of-the box policies and split-brain merging functionality for these new structures, split-brain merge configuration and SPI has been enhanced to ensure configuration and implementation is straightforward.
In IMDG 3.9 split-brain protection was available on IMap, ICache, IQueue and ILock. This has now been extended to other structures including ISet, IList, ISemaphore, ICountDownLatch, IReplicatedMap, IAtomicLong, IExecutorService, IDurableExecutorService, IScheduledExecutorServicem, IMultiMap, IRingBuffer, CardinalityEstimator (HyperLogLog), IAtomicReference and CRDT PN-Counter.
Generally, split-brain protection is more flexible in IMDG 3.10. Users can now apply different levels of severity to split-brain protection. Previously protection acted on information regarding the number of members currently observed. To improve this and provide better protection, split-brain protection now acts on additional information such as member heartbeats, membership changes, lost ICMP pings as well as successful pings following a lost ping. IMDG 3.10 includes numerous split-brain enhancements which provide better protection to structures during split-brain, consequently minimising inconsistent reads or writes.
For enterprise customers, IMDG 3.10 includes improved split-brain merging for HD structures and optimisations for merging large data sets, reducing the time HD structures take to merge after the cluster has reconciled. IMDG 3.10 also includes ICMP Failure Detector, a detector which works on ISO OSI Layer 3 and allows for the faster detection of dead members. Finally, IMDG 3.10 is the first IMDG on the market to be compliant with JCache 1.1.
Greg Luck, CEO of Hazelcast, said, “A major reason why we are adopted by open source projects and developers is because we listen to community feedback and act upon it. This new release is a testament to the strength of our community following and the quality advice we receive from it. This release brings the fastest split-brain detection, protection against mutation across almost all data structures during split brain and comprehensive healing across almost all data structures. To cap all that off, we have added some conflict-free data structures such as Flake IDs for ID generation, and CRDT Counters, as a conflict free alternative to AtomicLong.”
Comment on this article below or via Twitter @IoTGN