Azure Toolkit for IntelliJ – Spark Interactive Console

We are pleased to reveal the release of Spark Interactive Console in Azure Toolkit for IntelliJ. This new component facilitates Spark job authoring, and enables you to run code interactively in a shell-like environment within IntelliJ.

The Spark console includes Spark Local Console and Spark Livy Interactive Session. When you run the Spark console, instances of SparkSession and SparkContext are automatically instantiated like in Spark shell. You can use 'spark' to access the SparkSession and use 'sc' to access the SparkContext. The Spark local console allows you to run your code interactively and validate your code logic locally. You can also check your programming variables and perform other scripting operations locally before submitting to the cluster. The Spark Livy interactive session establishes an interactive communication channel with your cluster so you can check on file schemas, preview data, and run ad-hoc queries while you are programming your Spark job. You can also easily switch the Livy interactive session against different Spark clusters.

The Spark console has a Language Service built-in for Scala programming. You can leverage the language service features, such as IntelliSense and autocomplete, to look up a Spark object (i.e., Spark context and Spark session) properties, query hive metadata, and check on function signatures.

A new feature, Send Selection to Spark Console (Ctrl + Shift  + S), has been added to simplify the user experience for accessing the Spark console. You can send a highlighted single line of code or a block of code to the console from your main Scala project. This feature enables you to switch smoothly between contexts: coding and validation or testing code in the Spark console.

Summary of new features

◈ Run Spark local console
◈ Run Spark Livy interactive session console
◈ Language service for Scala enabled in the console
◈ Send selected code to console

The addition of the Spark console is an important step forward for the Azure Toolkit because of its expanding capabilities beyond batch job processing. This update also supports interactive querying across local and dev/test clusters.

To run your code and, please select and hold Ctrl + Enter, and use the up and down arrows to browse the history of previously run code.

How to access

You can easily start the Spark console either from the Tools menu or from the Scala file by right-clicking on the context menu.

Continue reading

Powerful Debugging Tools for Spark for Azure HDInsight

Microsoft runs one of the largest big data cluster in the world – internally called “Cosmos”. This runs millions of jobs across hundreds of thousands of servers over multiple Exabytes of data. Being able to run and manage jobs of this scale by developers was a huge challenge. Jobs with hundreds of thousands of vertices are common and to even quickly figure out why a job runs slow or narrow down bottlenecks was a huge challenge. We built powerful tools that graphically show the entire job graph including the various vertex execution times, playback etc. which helped developers greatly. While this was built for our internal language in Cosmos (called Scope), we are working very hard to bring this power to all Spark developers.

Today, we are delighted to announce the Public Preview of the Apache Spark Debugging Toolset for HDInsight for Spark 2.3 cluster and forward.   The default Spark history server user experience is now enhanced in HDInsight with rich information on your spark jobs with powerful interactive visualization of Job Graphs & Data Flows. The new features greatly assist HDInsight Spark developers in job data management, data sampling, job monitoring and job diagnosis.

Spark History Server Enhancements

The Spark History Server Experience in HDInsight now features two new tabs: Graph and Data.

Graph Tab – Job graph is a powerful interactive visualization of your jobs. This interface enables innovative debugging experiences such as playback, heatmap by progress of job stages, read, written for Spark application and individual jobs.

The Spark job graph displays Spark job executions details with data input and output across stages. For completed jobs, the Spark job graph allows Spark developer to playback the job by progress, data read and written with details. You can now dwell in Spark job diagnosis around performance, data and execution time using this experience which articulates various stage outliers.

Data Tab – Job specific input, output data view, search, download, preview, data copy, data URL copy, data export to CSV, as well as table operations view are visualized in the data tab.

As a developers or data scientist, you can perform various actions such as preview, download, copy and export to CSV file of the data. You can also come here to partially download data as sample data for your local run and local debug. Metadata interpretation and correlation has always been a challenge in debugging. A cool feature has also been added around Table Operations, you are able to view the Hive metadata, investigate table operations at each stage to gain more insights for better troubleshooting and spark job analysis.

Developer Nirvana

HDInsight Spark developers can greatly increase their productivity by leveraging these capabilities:

◈ Preview and download Spark job input and output data, as well as view Spark job table operations. 

◈ View and playback Spark application / job graph by progress, data read and written.

◈ Identify the Spark application execution dependencies among stages and jobs for performance tuning.

◈ View data read/write heatmap, identify the outliers and the bottlenecking stage and job for Spark job performance diagnosis.

◈ View Spark job/stage data input/output size and time duration for performance optimization.

◈ Locate failed stage and drill down for failed tasks details for debugging.

Getting Started with Apache Spark Debugging Toolset

These features have been built into HDInsight Spark history server.

Access from the Azure portal - Open the Spark cluster, click Cluster Dashboard from Quick Links, and then click Spark History Server.

More features to come

◈ Critical path analysis for Spark application and job

◈ Spark job diagnosis 

    ◈ Data Skew and Time Skew Analysis

    ◈ Executor Usage Analysis

◈ Debugging on failed job

Continue reading

Azure Toolkit for Eclipse integrates with HDInsight Ambari and supports Spark 2.2

To provide more authentication options, Azure Toolkit for Eclipse now supports integration with HDInsight clusters through Ambari for job submission, cluster resource browse and storage files navigate. You can easily link or unlink any cluster by using an Ambari-managed username and password, which is independent of your Azure sign-in credentials.  The Ambari connection applies to normal Spark and Hive hosted within HDInsight on Azure. These additions give you more flexibility in how you connect to your HDInsight clusters in addition to your Azure subscriptions while also simplifying your experiences in submitting Spark jobs.

With this release, you can benefit the new functionalities and consume the new libraries & APIs from Spark 2.2 in Azure Toolkit for Eclipse. You can create, author and submit a Spark 2.2 project to Spark 2.2 cluster.  With the backward compatibility of Spark 2.2, you can also submit your existing Spark 2.0 and Spark 2.1 projects to a Spark 2.2 cluster.

How to link a cluster

◈ Click Link a cluster from Azure Explorer.

◈ Enter Cluster Name, Storage Account, Storage Key, then select a container from Storage Container, at last, input Username and Password. Click the OK button to link cluster.

Please note that you can use either Ambari username, pwd or Secure Hadoop domain username, or pwd to connect. The storage account and key information will become optional in our next release for cluster connection.

◈ The Linked cluster is displayed under HDInsight node in Azure Explorer after linking. You can submit you Spark jobs to this linked cluster.

◈ You also can unlink a cluster from Azure Explorer.

How to install/update

Eclipse will prompt you for latest update if you have the plugin installed before, or you can get the latest bits by going to the Eclipse repository and searching Azure Toolkit for Java.

Continue reading

BigDL Spark deep learning library VM now available on Microsoft Azure Marketplace

BigDL deep learning library is a Spark-based framework for creating and deploying deep learning models at scale. While it has previously been deployed on Azure HDInsight and Data Science VM, it is now also available on Azure Marketplace as a fixed VM image, representing a further step by Intel to reduce deployment complexity for users.

Because BigDL is an integral part of Spark, a user does not need to explicitly manage distributed computations. A BigDL application provides high-level control “knobs”, such as the number of compute nodes, cores, and batch size, a BigDL application also leverages stable Spark infrastructure for node communications and resource management during its execution. BigDL applications can be written in either Python or Scala and achieve high performance through both algorithm optimization and taking advantage of close integration with Intel’s Math Kernel Library (MKL).

What is the Microsoft Azure Marketplace? The Azure Marketplace is an online applications and services marketplace that enables start-ups, independent software vendors (ISVs), and MSP/SIs to offer their Azure-based solutions or services to customers around the world. 

Introduction

This post describes two use case scenarios to deploy BigDL_v0.4 in Azure VMs:

◈ First scenario: Deploying an Azure VM with a pre-built BigDL_v0.4 image and running a basic deep learning example.

◈ Second scenario: Deploying BigDL_v0.4 on a bare-bones Ubuntu VM (for advanced users).

First scenario: Deploying a pre-built BigDL_v0.4 VM image:

Log in to your Microsoft Azure account. BigDL requires you to have an Azure subscription. You can get a free trial by visiting BigDL offering on Azure Marketplace and clicking Get it now.

You should see the following page. Click on the Create button at the bottom.

Enter the requested information in the fields at the prompts. Note that Azure imposes syntax limitations on some of the fields (such as using only alphanumeric characters and no CAPS). Use lowercase letters and digits and you will be fine. Use the following three screenshots for guidance.

Spark is memory-intensive, so select a machine with a larger amount of RAM. Note that not all VM types and sizes are available in certain regions. For simple tasks and testing, the virtual machine displayed in the following screenshot will meet requirements:

After the VM is provisioned, copy its public IP address. Note that this public IP address will change every time you stop and restart your VM. Keep this in mind if you are thinking of BigDL automation.

After deployment, you can modify the IP address provided in the resource group and set it up as a static IP address:

You are now ready to SSH into your BigDL VM. You can use your favorite SSH client. For this example, MobaXterm is used.

Enter the IP address and the username you selected when creating the VM.

Check the versions of installed dependencies:

Before using pre-installed BigDL, you will need to change ownership of the directory.

BigDL was pre-installed into the bigdlazrmktplc directory. Yet ‘testuser’ does not have full privileges to access it.

To change this, type:

$sudo chown -R testuser:testuser bigdlazrmktplc

Now ‘testuser’ owns the bigdlazrmktplc directory.

Finally, test that BigDL actually works in this VM by entering the following commands:

$cd bigdlazrmktplc/BigDL

$export SPARK_HOME=/usr/local/spark/spark-2.2.0-bin-hadoop2.7

$export BIGDL_HOME=/home/bigdlazrmktplc/BigDL

$BigDL/bin/pyspark-with-bigdl.sh --master local[*]

If the commands are successful you will see the following:

At the command prompt, copy and paste the following example code, the source can be found on GitHub

from bigdl.util.common import *

from pyspark import SparkContext

from bigdl.nn.layer import *

import bigdl.version

# create sparkcontext with bigdl configuration

sc = SparkContext.getOrCreate(conf=create_spark_conf().setMaster("local[*]"))

init_engine() # prepare the bigdl environment

bigdl.version.__version__ # Get the current BigDL version

linear = Linear(2, 3) # Try to create a Linear layer

If the commands are successful, you will see the following:

BigDL is now ready for you to use.

Second Scenario: Deploying BigDL_v0.4 on a bare-bones Ubuntu Azure VM

First, you will need to create an Azure subscription. You can get a free trial by navigating to BigDL offering on Azure Marketplace and clicking Get it now.

Log in to the Azure Portal, go to New, and select Ubuntu server 16.04 LTS VM (LTS = Long Term Support).

Enter the basic VM attributes using only lower-case letters and numbers.

For Spark jobs you want to select VMs with a large amount of RAM available.

After your VM has been created, you can SSH into it using the username and password which you created previously.

Copy the Public IP address of the VM:

This creates a very basic Ubuntu machine, so you must install the following addtional components to run BigDL:

◈ Java Runtime Environment (JRE)

◈ Scala

◈ Spark

◈ Python packages

◈ BigDL

Installing the Java Runtime Environment (JRE)

At the command prompt, type the following commands:

$sudo add-apt-repository ppa:webupd8team/java

$sudo apt-get update

$sudo apt-get install oracle-java8-installer

$sudo apt-get install oracle-java8-set-default

Confirm the installation and JRE version by typing

$java -version

Installing Scala and confirming version

At the command prompt, type the following commands:

$sudo apt-get install scala

$scala -version

Installing Spark 2.2.x

At the command prompt, type the following commands:

$sudo wget http://mirrors.ibiblio.org/apache/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz

$sudo tar xvzf spark-2.2.0-bin-hadoop2.7.tgz

$rm spark-2.2.0-bin-hadoop2.7.tgz

$sudo mkdir /usr/local/spark

$sudo mv spark-2.2.0-bin-hadoop2.7 /usr/local/spark

Verify Spark installation:

$cd  /usr/local/spark/spark-2.2.0-bin-hadoop2.7/

$./bin/spark-submit –version

Installing BigDL

The main repo for BigDL downloadable releases.

For Spark 2.2.0 and Scala 2.11.x, select Dist-spark-2.2.0-scala-2.11.8-all-0.4.0-dist.zip

At the command prompt, type the following commands:

$cd ~

$mkdir BigDL

$cd BigDL

$sudo wget https://s3-ap-southeast-1.amazonaws.com/bigdl-download/dist-spark-2.2.0-scala-2.11.8-all-0.4.0-dist.zip

 $sudo apt-get install unzip

 $unzip dist-spark-2.2.0-scala-2.11.8-all-0.4.0-dist.zip

 $rm dist-spark-2.2.0-scala-2.11.8-all-0.4.0-dist.zip

Installing Python 2.7 packages

Ubuntu 16x on Azure comes with pre-installed python 2.7. However, there are a couple of additional packages that must be installed.

At the command prompt, type the following commands:

$sudo apt-get install python-numpy

$sudo apt-get install python-six

Update all packages and dependencies by typing

$sudo apt-get update

Verify BigDL installation

The instructions to verify that BigDL was installed correctly are available.

At the command prompt, type the following commands:

$export SPARK_HOME=/usr/local/spark/spark-2.2.0-bin-hadoop2.7

$export BIGDL_HOME=/home/bigdlazrmktplc/BigDL

Launch PySpark (from BigDL directory)

$bin/pyspark-with-bigdl.sh --master local[*]

At the prompt, copy and paste the following code, this code can also be found at Github.

from bigdl.util.common import *

from pyspark import SparkContext

from bigdl.nn.layer import *

import bigdl.version

# create sparkcontext with bigdl configuration

sc = SparkContext.getOrCreate(conf=create_spark_conf().setMaster("local[*]"))

init_engine() # prepare the bigdl environment

bigdl.version.__version__ # Get the current BigDL version

linear = Linear(2, 3) # Try to create a Linear layer

You should see the following:

creating: createLinear

cls.getname: com.intel.analytics.bigdl.python.api.Sample

BigDLBasePickler registering: bigdl.util.common  Sample

cls.getname: com.intel.analytics.bigdl.python.api.EvaluatedResult

BigDLBasePickler registering: bigdl.util.common  EvaluatedResult

cls.getname: com.intel.analytics.bigdl.python.api.JTensor

BigDLBasePickler registering: bigdl.util.common  JTensor

cls.getname: com.intel.analytics.bigdl.python.api.JActivity

BigDLBasePickler registering: bigdl.util.common  JActivity

>>>

Finally, install Maven to allow you to build BigDL applications by typing the following:

$sudo apt-get install maven

Your VM is now ready for running deep learning examples at scale.

Continue reading