News

Open Source Framework Enables Streaming Data Pipelines on Kubernetes

• By John K. Waters
• January 30, 2020

Lightbend, the company behind the Scala JVM language and developer of the Reactive Platform, recently launched an open source framework for developing, deploying, and operating streaming data pipelines on Kubernetes. The Cloudflow framework, hosted on cloudflow.io, was developed to address the growing demands of AI, machine learning models, analytics, and other streaming, data-driven workloads.

"The most interesting new opportunities around AI and training machine learning models are putting a lot of pressure on developers to get streaming data into new applications," said Lightbend CEO Mark Brewer in a statement, "and to have predictability on the uninterrupted flow of that data once in production."

Cloudflow was designed to make streaming data pipelines easy to build, deploy, and integrate as fundamental building blocks for cloud-native applications, Brewer said.

"Cloudflow allows you to easily break down your streaming application into smaller components and wire them together with schema-based contracts," explained Craig Blitz, Lightbend senior product director, in a blog post.

Cloudflow comprises a set of tools for app development, including:

• An API definition for Streamlet, the core abstraction in Cloudflow.
• An extensible set of runtime implementations for Streamlet(s). (It currently supports several streaming runtimes, including Spark Structured Streaming, Flink, and Lightbend's own Akka Streams.)
• A Streamlet composition model driven by a blueprint definition.
• A sandbox execution mode that accelerates the development and testing of an application.
• A set of sbt plugins that package an application into a deployable container.
• The Cloudflow operator, which is a Kubernetes operator that manages the application lifecycle.
• A CLI, in the form of a kubectl plugin that facilitates manual and scripted management of the application.

The Cloudflow operator takes care of orchestrating the deployment of the different parts of a streaming pipeline as an end-to-end application on the Kubernetes cluster. The result, the company says, is a dramatic reduction in the time required to create, package, and deploy an application from weeks to hours.

Lightbend also offers support and enhanced deployment and operational features with a Lightbend Subscription.