News

JetBrains Plugs IntelliJ IDEA Into Big Data

• By John K. Waters
• June 1, 2020

Software development toolmaker JetBrains, creator of the Java dev fav IntelliJ IDEA, today unveiled the full public access preview of Big Data Tools, a new plugin designed to allow developers to work with Zeppelin notebooks, Spark applications, and S3 files from within that popular IDE. Previously available as a preview in an early access program for IntelliJ IDEA Ultimate users, the plugin is now also available to users of the company's DataGrip database IDE and PyCharm Professional Python IDE.

For Zeppelin notebooks written in Scala and Spark SQL, the plugin offers smart navigation, code completion, inspections and quick-fixes, and refactoring inside the notebooks. It is possible to browse, create, and run Zeppelin notebooks without leaving the IDE, the company says. The plugin also allows users to track Spark jobs, see cluster environment and storage information from the IDE, and browse and manage files in HDFS, S3, and other supported storage solutions. The tool also supports columnar storage solutions, such as Parquet, which have not been available until now, the company says.

The Prague-based company is billing Big Data Tools as the first product to provide synergy among all the tools built on top of the IntelliJ Platform. It's a kind of acknowledgement of the impact of Big Data on developers -- including some significantly changing roles -- and the need for "the best possible tools for developers, data scientists, and engineers."

"The adoption of Big Data across industries continues to accelerate alongside demand for the right tooling from the enterprise," said Vitaly Khudobakhshov, JetBrains' Big Data Tools project lead, in a statement. "We believe JetBrains can leverage the developer platform we have already to build this tooling, empowering people all over the world to build Big Data solutions faster and enjoy the process."

A lot of data engineers who use Apache Zeppelin for Scala already use IntelliJ IDEA for Scala and Java code, Khudobakhshov explained. So the integration between Apache Zeppelin and IntelliJ IDEA was natural; it allows data engineers to work with Zeppelin notebooks, Apache Spark applications, and Amazon S3 files without having to leave the IDE. The company argues that this is an increasingly important capability for data scientists, "as there are a lot of scenarios when it's more convenient to use data tools through the native IDE interface."

JetBrains' plans move into this space is a real commitment, the company said in a statement, and includes plans "to create tools to cover all essential scenarios for data engineers" with the new Big Data Tools offering.

The company released IntelliJ IDEA 2020.1, a major update of its flagship code-centric Java IDE, in April. The first major update released this year, it arrived with with support for the latest Java 14 release, as well as new features for several Web and test frameworks, an upgrade of the debugger with dataflow analysis assistance, and a new LightEdit mode. The company's newest product is Space, an all-in-one team collaboration environment.