News

Hazelcast Enhances Real-Time Stream Processing Engine

• By John K. Waters
• June 14, 2017

In-memory data grid (IMDG) specialist Hazelcast Inc. yesterday the release of an updated version of its relatively new distributed processing engine for Big Data streams. Hazelcast Jet 0.4 comes with several new capabilities, including event-time processing with tumbling, sliding and session windowing.

The Palo Alto, Calif.-based company unveiled the open source, Apache 2-licensed Jet in February. The new processing engine is designed to process data in parallel across nodes, enabling data-intensive applications to operate in near real-time. It's built on top of a one-record-per-time architecture, sometimes known as the continuous operator model. This approach processes incoming records as soon as possible, as opposed to accumulating records into micro-batches. The result is lowering latency for applications, the company said.

The new windowing functionality allows users to evaluate stream processing jobs at regular time intervals, regardless of how many incoming messages the job is processing, the company says. The new release provides three types of windows: fixed/tumbling, in which time is partitioned into same-length, non-overlapping chunks, and each event belongs to exactly one window; sliding, in which windows have a fixed length, but are separated by a time interval that can be smaller than the window length; and session, in which windows have various sizes and are defined based on data, which should carry some session identifiers.

The company is underscoring the benefits of these new capabilities for developers working with IoT architectures requiring sensor updates -- things like house thermostats and lighting systems-- as well as in-store e-commerce systems and social media platforms. Jet supports both stream and batch processing, but stream is quickly surpassing batch as a preferred method of processing big data sets for companies that require immediate insight into data, the company says.

Of special interest to Java developers in this release is Jet's new ability to use ICache/Hazelcast integration as a source and sink of data. java.util.stream can now be used on top of ICache to enable basic data processing.

Hazelcast has roots planted deeply in the Java language and platform (its IMDG is written in Java), but the company has spread its support over the years, thanks in part to the efforts of its open source community, to include several clients and programming languages, including .NET/C++, Node.js, Python, Clojure and Scala.

"The new functionality in 0.4 brings stream processing for the first time," said Hazelcast CEO Greg Luck, in a statement.