News

Microsoft Borrows from Visual, Low-Code Approach in Azure Data Factory

• By David Ramel
• January 24, 2018

Microsoft has shipped a v2 preview of its Azure Data Factory -- a cloud-based, Big Data integration service -- that borrows from the visual, low-code development approach that has seen skyrocketing popularity of late.

The drag-and-drop functionality was injected into Azure Data Factory (ADF) v2 in response to developer feedback collected since it debuted in version 1 in October 2015, Microsoft said.

ADF is used to create data-driven workflows -- such as extract, transform, load (ETL) and extract, load, transform (ELT) -- in the cloud in order to orchestrate and automate data movement and data transformation.

Processing and transforming the data is done in conjunction with compute services such as Azure HDInsight Hadoop, Spark, Azure Data Lake Analytics and Azure Machine Learning.

One of ADF's main uses is the creation and scheduling of those data-driven workflows (Microsoft calls them "pipelines": logical groupings of activities that together perform a task) that can ingest data from disparate data stores.

That process is now point-and-click simple in the v2 preview.

"One of the consistent pieces of customer feedback we received, is to enable a rich interactive visual authoring and monitoring experience allowing users to create, configure, test, deploy and monitor data integration pipelines without any friction," said Microsoft's Gaurav Malhotra, senior program manager, in a blog post last week.

"We listened to your feedback and are happy to announce the release of visual tools for ADF v2. The main goal of the ADF visual tools is to allow you to be productive with ADF by getting pipelines up & running quickly without requiring [you] to write a single line of code."

The new visual functionality lets developers create control flow pipelines, drag-and-drop activities onto a canvas and "connect them on-success, on-failure, on-completion." Visual monitoring of pipeline, activity and trigger runs is also available in a simple list view interface.

ADF v2 also includes guided tours that help developers understand how to use the new visual authoring and monitoring features while also providing a mechanism to give the ADF team more valuable feedback.

See Malhotra's post for more details on the above functionality and much more. Additional guidance is provided in the Jan. 16 article "Create a data factory using the Azure Data Factory UI," which covers the v2 preview.