Azure Toolkit for Eclipse integrates with HDInsight Ambari and supports Spark 2.2

To provide more authentication options, Azure Toolkit for Eclipse now supports integration with HDInsight clusters through Ambari for job submission, cluster resource browse and storage files navigate. You can easily link or unlink any cluster by using an Ambari-managed username and password, which is independent of your Azure sign-in credentials.  The Ambari connection applies to normal Spark and Hive hosted within HDInsight on Azure. These additions give you more flexibility in how you connect to your HDInsight clusters in addition to your Azure subscriptions while also simplifying your experiences in submitting Spark jobs.

With this release, you can benefit the new functionalities and consume the new libraries & APIs from Spark 2.2 in Azure Toolkit for Eclipse. You can create, author and submit a Spark 2.2 project to Spark 2.2 cluster.  With the backward compatibility of Spark 2.2, you can also submit your existing Spark 2.0 and Spark 2.1 projects to a Spark 2.2 cluster.

How to link a cluster

◈ Click Link a cluster from Azure Explorer.

◈ Enter Cluster Name, Storage Account, Storage Key, then select a container from Storage Container, at last, input Username and Password. Click the OK button to link cluster.

Please note that you can use either Ambari username, pwd or Secure Hadoop domain username, or pwd to connect. The storage account and key information will become optional in our next release for cluster connection.

◈ The Linked cluster is displayed under HDInsight node in Azure Explorer after linking. You can submit you Spark jobs to this linked cluster.

◈ You also can unlink a cluster from Azure Explorer.

How to install/update

Eclipse will prompt you for latest update if you have the plugin installed before, or you can get the latest bits by going to the Eclipse repository and searching Azure Toolkit for Java.