Traffic Analytics now generally available

We are excited to announce the general availability (GA) of the Traffic Analytics, a SaaS solution that provides visibility into user and application traffic on your cloud networks.

The solution has analyzed several terabytes of Flow logs on a regular basis for network activity across virtual subnets, VNets, Azure data center regions and VPNs, and provided actionable insights that helped our customers:

◈ Audit their networks and root out shadow-IT and non-compliant workloads.
◈ Optimize the placement of their workloads and improve the user experience for their end users.
◈ Detect security issues and improve application and data security.
◈ Reduce costs and right size their deployments by eliminating the issue of over-provisioning or under-utilization.
◈ Gain visibility into their public cloud networks spanning multiple Azure regions across numerous subscriptions.

This GA release includes enhancements that help you detect issues and secure/optimize your network, faster and more intuitively than before.

Some of the enhancements in this release are:

◈ Your environment: Provides a view into your entire Azure network, identifies inactive regions, virtual networks, and subnets – for example, network locations with VMs and no network activity for further analysis. Detects malicious flows as they flow across application gateways, subnets, and networks. Indicate open ports conversing over the Internet and hosts sending traffic to the Internet to qualify possible threats.

Figure 1: Your environment provides an overview of your cloud network with drill-down into regions, VNets, and subnets with intuitive and visually rich network maps.

◈ Summary view: Provides a summary of allowed, blocked, benign, and malicious flows across inbound and outbound traffic. Unusual increase in traffic types merit forensic investigations, such as higher number of allowed malicious flows and higher number of benign blocked flows.

Figure 2: Traffic Summary highlights the allowed malicious flows and large number of benign, but blocked, flows.

◈ Application activity: Identify workload activity including applications generating or consuming the most flows and the top VM conversation pairs at granularities ranging from VNets to hosts. Secure your network using insights from malicious and blocked traffic by the application/port, or update your network security groups (NSG) to allow normal traffic. For example, identify which ports are open for communication and allowing malicious flows. These merit further investigation and possible update in NSG configuration.

Figure 3: Traffic flow distribution at host, subnet, and VNet granularity.

Figure 4: Flow by port determines the top applications on the network and the top consumers.

◈ Capacity planning: VPNs constitute an important medium for hybrid and inter-VNet connectivity. View utilization across your gateways, and detect under-utilized or maxed-out gateways. Use the list of top VPN connections per gateway to understand your traffic patterns, distribute  traffic load, and eliminate downtime due to under provisioning.

Figure 5: Capacity utilization of VPN gateways.

◈ Application Gateway and Load Balancer support: Traffic Analytics now extends its analytics capabilities to include traffic flowing through Azure Application Gateways and Load Balancers. Get insights on traffic pattern, resources impacted by malicious traffic, and traffic distribution to backend pool instances and hosts.

Figure 6: Flow statistics for Application Gateways and Load Balancers.

◈ Secure your cloud network with NSG insights: Gain detailed statistics ranging from the top five NSGs and NSG rules to detailed flow information, allowing you to answer the questions, “How effective are your NSGs?”, “What are the top rules per NSG?”, “Are they allowing traffic from malicious sources?”, “What are the flow statistics per NSG?”, “What are the top talking pairs per NSG?”, and more.

Figure 7: Detailed statistics on NSGs and time series charts.

◈ Automate your deployment: Have several NSGs across regions that need to be enabled for analysis? Traffic Analytics now supports PowerShell (v6.2.1 and higher) to get you up and analyzing in minutes.

◈ More regions: You can now add a workspace in South East Asia and/or analyze NSGs in this Azure region.

Continue reading

Network Performance Monitor’s Service Connectivity Monitor is now generally available

Network Performance Monitor’s (NPM) Service Connectivity Monitor, previously in preview as Service Endpoint Monitor, is making general availability sporting a new name. With Service Connectivity Monitor, you can now monitor connectivity to services such as applications, URIs, VMs, and network devices, as well as determine what infrastructure is in the path and where network bottlenecks are occurring.

As services and users are becoming more dispersed across clouds, branch offices, and remote geographies, it is becoming more difficult to determine the cause of a service outage or performance degradation. These can be due to an issue with the application, stack, or cluster as well as network issues in the cloud, the carrier network, or in the first-mile. Service Connectivity Monitor integrates the monitoring and visualization of the performance of your internally hosted and cloud services with the end-to-end network performance. You can create connectivity tests from key points in your network to your services and identify whether the problem is due to the network or the application. With the network topology map, you can locate the links and interfaces experiencing high loss and latencies, helping you identify external and internal troublesome network segments.

Determine if it’s an application or a network problem

You can determine whether the application connectivity issue is due to the application or the network by corelating the application response time with the network latency.

The example image below illustrates a scenario where spikes in the application response time are accompanied with corresponding spikes in the network latency. This suggests that the application degradation is due to an increase in network latency, and therefore, the issue is due to the underlying network, and not the application.

The example snippet below demonstrates another scenario where there is a spike in the application response time whereas the network latency is consistent. This suggests that the network was in a steady state, when the performance degradation was observed. Therefore, the problem is due to an issue at the application end.

Identify network bottlenecks

You can view all the paths and interfaces between your corporate premises and application endpoint on NPM’s interactive topology map. You not only get end-to-end network visibility from your nodes to the application, but you can also view the latency contributed by each interface to help you identify the troublesome network segment. The image below illustrates a scenario where you can identify the highlighted network interface as the one causing most latency.

Monitor service connectivity from multiple vantage points from a central view

You can monitor connectivity to your services from your branch offices, datacenters, office sites, cloud infrastructure, etc. from a central view. By installing the NPM agents at the vantage points in your corporate perimeter, you can get the performance visibility from where your users are accessing the application.

The below example image illustrates a scenario where you can get the network topology from multiple source nodes to www.msn.com in a single pane of view and identify the nodes with connectivity issues from the unhealthy paths in red.

Monitor end-to-end connectivity to services

Monitor the total response time, network latency, and packet loss between the source nodes in your corporate perimeter and the services you use, such as websites, SaaS, PaaS, Azure services, file servers, SQL servers, among others. You can setup alerts to get proactively notified whenever the response time, loss, or latency from any of your branch offices crosses the threshold. In addition to viewing the near real-time values and historical trends of the performance data, you can use the network state recorder to go back in time to view a particular network state in order to investigate the difficult-to-catch transient issues.

Monitor connectivity to Microsoft services using built-in tests for Microsoft Office 365 and Dynamics 365

Service Connectivity Monitor provides built-in tests that allow a simple one-click setup experience to monitor connectivity to Microsoft’s Office 365 and Dynamics 365 services, without any pre-configuration. Since the capability maintains a list of endpoints associated with these services, you do not have to enter the various endpoints associated with each service. 

Create custom queries and views

All data that is exposed graphically through NPM’s UI are also available natively in Log Analytics search. You can perform interactive analysis of data in the repository, correlate data from different sources, create custom alerts and views, and export the data to Excel, PowerBI, or a shareable link.

Continue reading

Structured streaming with Azure Databricks into Power BI & Cosmos DB

In this blog we’ll discuss the concept of Structured Streaming and how a data ingestion path can be built using Azure Databricks to enable the streaming of data in near-real-time. We’ll touch on some of the analysis capabilities which can be called from directly within Databricks utilising the Text Analytics API and also discuss how Databricks can be connected directly into Power BI for further analysis and reporting. As a final step we cover how streamed data can be sent from Databricks to Cosmos DB as the persistent storage.

Structured streaming is a stream processing engine which allows express computation to be applied on streaming data (e.g. a Twitter feed). In this sense it is very similar to the way in which batch computation is executed on a static dataset. Computation is performed incrementally via the Spark SQL engine which updates the result as a continuous process as the streaming data flows in.

The above architecture illustrates a possible flow on how Databricks can be used directly as an ingestion path to stream data from Twitter (via Event Hubs to act as a buffer), call the Text Analytics API in Cognitive Services to apply intelligence to the data and then finally send the data directly to Power BI and Cosmos DB.

The concept of structured streaming

All data which arrives from the data stream is treated as an unbounded input table. For each new data within the data stream, a new row is appended to the unbounded input table. The entirety of the input isn’t stored, but the end result is equivalent to retaining the entire input and executing a batch job.

The input table allows us to define a query on itself, just as if it were a static table, which will compute a final result table written to an output sink. This batch-like query is automatically converted by Spark into a streaming execution plan via a process called incremental execution.

Incremental execution is where Spark natively calculates the state required to update the result every time a record arrives. We are able to utilize built in triggers to specify when to update the results. For each trigger that fires, Spark looks for new data within the input table and updates the result on an incremental basis.

Queries on the input table will generate the result table. For every trigger interval (e.g. every three seconds) new rows are appended to the input table, which through the process of Incremental Execution, update the result table. Each time the result table is updated, the changed results are written as an output.

The output defines what gets written to external storage, whether this be directly into the Databricks file system, or in our example CosmosDB.

To implement this within Azure Databricks the incoming stream function is called to initiate the StreamingDataFrame based on a given input (in this example Twitter data). The stream is then processed and written as parquet format to internal Databricks file storage as shown in the below code snippet:

val streamingDataFrame = incomingStream.selectExpr("cast (body as string) AS Content")

.withColumn("body", toSentiment(%code%nbsp;"Content"))

import org.apache.spark.sql.streaming.Trigger.ProcessingTime

val result = streamingDataFrame

.writeStream.format("parquet")

.option("path", "/mnt/Data")

.option("checkpointLocation", "/mnt/sample/check")

.start()

Mounting file systems within Databricks (CosmosDB)

Several different file systems can be mounted directly within Databricks such as Blob Storage, Data Lake Store and even SQL Data Warehouse. In this blog we’ll explore the connectivity capabilities between Databricks and Cosmos DB.

Fast connectivity between Apache Spark and Azure Cosmos DB accelerates the ability to solve fast moving Data Sciences problems where data can be quickly persisted and retrieved using Azure Cosmos DB. With the Spark to Cosmos DB connector, it’s possible to solve IoT scenarios, update columns when performing analytics, push-down predicate filtering, and perform advanced analytics against fast changing data against a geo-replicated managed document store with guaranteed SLAs for consistency, availability, low latency, and throughput.

◈ From within Databricks, a connection is made from the Spark master node to Cosmos DB gateway node to get the partition information from Cosmos.

◈ The partition information is translated back to the Spark master node and distributed amongst the worker nodes.

◈ That information is translated back to Spark and distributed amongst the worker nodes.

◈ This allows the Spark worker nodes to interact directly to the Cosmos DB partitions when a query comes in. The worked nodes are able to extract the data that is needed and bring the data back to the Spark partitions within the Spark worker nodes.

Communication between Spark and Cosmos DB is significantly faster because the data movement is between the Spark worker nodes and the Cosmos DB data nodes.

Using the Azure Cosmos DB Spark connector (currently in preview) it is possible to connect directly into a Cosmos DB storage account from within Databricks, enabling Cosmos DB to act as an input source or output sink for Spark jobs as shown in the code snippet below:

import com.microsoft.azure.cosmosdb.spark.CosmosDBSpark

import com.microsoft.azure.cosmosdb.spark.config.Config

val writeConfig = Config(Map("Endpoint, MasterKey, Database, PreferredRegions, Collection, WritingBatchSize"))

import org.apache.spark.sql.SaveMode

sentimentdata.write.mode(SaveMode.Overwrite).cosmosDB(writeConfig)

Connecting Databricks to PowerBI

Microsoft Power BI is a business analytics service that provides interactive visualizations with self-service business intelligence capabilities, enabling end users to create reports and dashboards by themselves without having to depend on information technology staff or database administrators.

Azure Databricks can be used as a direct data source with Power BI, which enables the performance and technology advantages of Azure Databricks to be brought beyond data scientists and data engineers to all business users.

Power BI Desktop can be connected directly to an Azure Databricks cluster using the built-in Spark connector (Currently in preview). The connector enables the use of DirectQuery to offload processing to Databricks, which is great when you have a large amount of data that you don’t want to load into Power BI or when you want to perform near real-time analysis as discussed throughout this blog post.

This connector utilises JDBC/ODBC connection via DirectQuery, enabling the use of a live connection into the mounted file store for the streaming data entering via Databricks. From Databricks we can set a schedule (e.g. every 5 seconds) to write the streamed data into the file store and from Power BI pull this down regularly to obtain a near-real time stream of data.

From within Power BI, various analytics and visualisations can be applied to the streamed dataset bringing it to life!

Want to have a go at building this architecture out? For more examples of Databricks see the official Azure documentation:

◈ Perform ETL operations in Databricks.

◈ Structured Streaming in Databricks.

◈ Stream Data from HDInsight Kafka.

Continue reading

Traffic Analytics now generally available

We are excited to announce the general availability (GA) of the Traffic Analytics, a SaaS solution that provides visibility into user and application traffic on your cloud networks.

◈ Audit their networks and root out shadow-IT and non-compliant workloads.
◈ Optimize the placement of their workloads and improve the user experience for their end users.
◈ Detect security issues and improve application and data security.
◈ Reduce costs and right size their deployments by eliminating the issue of over-provisioning or under-utilization.
◈ Gain visibility into their public cloud networks spanning multiple Azure regions across numerous subscriptions.

This GA release includes enhancements that help you detect issues and secure/optimize your network, faster and more intuitively than before.

Some of the enhancements in this release are:

◈ Your environment: Provides a view into your entire Azure network, identifies inactive regions, virtual networks, and subnets – for example, network locations with VMs and no network activity for further analysis. Detects malicious flows as they flow across application gateways, subnets, and networks. Indicate open ports conversing over the Internet and hosts sending traffic to the Internet to qualify possible threats.

Figure 1: Your environment provides an overview of your cloud network with drill-down into regions, VNets, and subnets with intuitive and visually rich network maps.

◈ Summary view: Provides a summary of allowed, blocked, benign, and malicious flows across inbound and outbound traffic. Unusual increase in traffic types merit forensic investigations, such as higher number of allowed malicious flows and higher number of benign blocked flows.

Figure 2: Traffic Summary highlights the allowed malicious flows and large number of benign, but blocked, flows.

◈ Application activity: Identify workload activity including applications generating or consuming the most flows and the top VM conversation pairs at granularities ranging from VNets to hosts. Secure your network using insights from malicious and blocked traffic by the application/port, or update your network security groups (NSG) to allow normal traffic. For example, identify which ports are open for communication and allowing malicious flows. These merit further investigation and possible update in NSG configuration.

Figure 3: Traffic flow distribution at host, subnet, and VNet granularity.

Figure 4: Flow by port determines the top applications on the network and the top consumers.

◈ Capacity planning: VPNs constitute an important medium for hybrid and inter-VNet connectivity. View utilization across your gateways, and detect under-utilized or maxed-out gateways. Use the list of top VPN connections per gateway to understand your traffic patterns, distribute  traffic load, and eliminate downtime due to under provisioning.

Figure 5: Capacity utilization of VPN gateways.

◈ Application Gateway and Load Balancer support: Traffic Analytics now extends its analytics capabilities to include traffic flowing through Azure Application Gateways and Load Balancers. Get insights on traffic pattern, resources impacted by malicious traffic, and traffic distribution to backend pool instances and hosts.

Figure 6: Flow statistics for Application Gateways and Load Balancers.

◈ Secure your cloud network with NSG insights: Gain detailed statistics ranging from the top five NSGs and NSG rules to detailed flow information, allowing you to answer the questions, “How effective are your NSGs?”, “What are the top rules per NSG?”, “Are they allowing traffic from malicious sources?”, “What are the flow statistics per NSG?”, “What are the top talking pairs per NSG?”, and more.

Figure 7: Detailed statistics on NSGs and time series charts.

◈ Automate your deployment: Have several NSGs across regions that need to be enabled for analysis? Traffic Analytics now supports PowerShell (v6.2.1 and higher) to get you up and analyzing in minutes.
◈ More regions: You can now add a workspace in South East Asia and/or analyze NSGs in this Azure region.

Continue reading

Enabling Smart Manufacturing with Edge Computing

Smart Manufacturing envisions a future where factory equipment can make autonomous decisions based on what’s happening on the factory floor. Businesses can more easily integrate all steps of the manufacturing process including design, manufacturing, supply chain and operation. This facilitates greater flexibility and reactivity when participating in competitive markets. Enabling this vision requires a combination of related technologies such as IoT, AI/machine learning, and Edge Computing. In this article, we will introduce Edge Computing and discuss its role in enabling Smart Manufacturing.

What is Edge Computing?

Put simply, Edge Computing is about taking code that runs in the cloud and running it on local devices or close to it. Like in a gateway device or a PC sitting next to the device.

To understand Edge Computing it helps to think of an IoT solution as generally having three components:

◈ Things like IoT devices, which generate sensor data.
◈ Insights you extract from this data.
◈ Actions you perform based on these insights to deliver some sort of value.

With Edge Computing, you move the insights and actions components from the cloud to the device. In other words, you bring some of the code used to process and extract insights from the data, and perform some action in response to it right into the device.

Edge Computing model

You may still want to send data to the cloud for further processing and actions, but now you have the option of balancing your workloads more efficiently based on latency, compute power, or data privacy requirements. Notably, one of the things we can move to the edge is machine learning and AI algorithms. When we do this, we say that we have an intelligent edge.

When is Edge Computing useful?

Edge computing is useful in the following situations frequently found in manufacturing:

◈ Mission-critical scenarios where you need real-time insights for quick decision making. Basically, when the electrons cannot travel fast enough to the cloud and back, to do what you need to do. Think about an autonomous vehicle approaching a red light. It can’t afford the second delay to route the message to the cloud and wait for a command back. When the car sees a red light, it needs to perform analytics in a closed loop and stop immediately. Increasingly, these insights are being provided by machine learning models running on the device, performing predictive analytics, anomaly detection, image recognition, and classification.

◈ Remote sites where cloud connectivity is not stable or economical. For example, remote oil fields send data to the cloud only twice a day via satellite connection.

◈ Geographies where compliance and data residency are critical. You can pre-process sensitive data locally within a sovereign or organizational boundary.

◈ Avoiding transferring terabytes of raw data between devices and the cloud to reduce bandwidth costs. You can perform data cleaning and aggregation locally.

Remote oil fields are prime candidates for Edge Computing

How does Edge Computing compare to private cloud?

Private cloud is essentially a model of cloud where IT services are provisioned over private infrastructure for the dedicated use of the organization. Private cloud is best suited where you have cloud applications or application components that need to be deployed on-premises to overcome latency, connectivity, or regulatory requirements like when you need to isolate your environment from the public cloud.

Edge Computing, on the other hand, is about running computations on the device, and communicating to the cloud when needed. Now, the cloud these devices communicate with can be the public or private cloud. The device doesn’t really care.

Edge and private cloud can be operating in the same environment. A good example would be a cruise ship where passenger services and navigation apps would run on the private cloud, and engine maintenance would run on the edge.

Now, to make things more interesting, there is also the option of running a hybrid cloud. Which is a combination of private and public. In fact, many companies run their cloud environments this way because it gives them the best of both worlds. The latency and privacy of the private cloud, with the scalability and economies of scale of the public cloud. So, as you can imagine, you can have scenarios where you are running a hybrid cloud environment, with some things running in the public cloud and some things running in the private cloud. In addition, your IoT devices are running Edge Computing with some things running on the devices and some things running on the cloud.

Hybrid Cloud and Edge Computing Scenario

How does Edge Computing enable Smart Manufacturing?

The smart part of smart factory is about autonomy. The ability of some asset or piece of equipment to make decisions based on what’s going on in the factory floor without the need of human intervention, these are decisions that are not necessarily pre-programmed in some factory control system. The aim of Smart Manufacturing is to utilize a more programmatic data-led approach to develop new and higher quality goods faster. Edge Computing can enable this autonomy where machines in the factory floor extract insight and formulate actions at near real-time. Running AI/machine learning algorithms in their own electronics, almost as if they had their own brains!

Consider this example where you have two robots that are performing some task. The robots are connected to an edge device that is running a machine learning model listening to sensor data from the robots and whose mission is to predict an impending failure.

When the machine learning model determines that one of the robots is about to fail, it triggers actions. These actions may include stopping or slowing down the line that feeds the robot in trouble, and/or triggering a process in the cloud to create a service request in some line-of-business application.

What does Azure offer to enable Edge Computing?

The service Azure offers to enable Edge Computing is called Azure IoT Edge. IoT Edge is made up of three components:

◈ Azure IoT Edge modules are containers that run Azure services, 3rd party services, or your own code. They are deployed to IoT Edge devices and execute locally on those devices.

◈ The Azure IoT Edge runtime runs on each IoT Edge device and manages the modules deployed to each device. An interesting fact is this runtime will be open sourced to the developer community so that they can make changes and additions to it.

◈ A cloud-based interface enables you to remotely monitor and manage IoT Edge devices.

Continue reading

Siphon: Streaming data ingestion with Apache Kafka

Data is at the heart of Microsoft’s cloud services, such as Bing, Office, Skype, and many more. As these services have grown and matured, the need to collect, process and consume data has grown with it as well. Data powers decisions, from operational monitoring and management of services, to business and technology decisions. Data is also the raw material for intelligent services powered by data mining and machine learning.

Most large-scale data processing at Microsoft has been done using a distributed, scalable, massively parallelized storage and computing system that is conceptually similar to Hadoop. This system supported data processing using a batch processing paradigm. Over time, the need for large scale data processing at near real-time latencies emerged, to power a new class of ‘fast’ streaming data processing pipelines.

Siphon – an introduction

Siphon was created as a highly available and reliable service to ingest massive amounts of data for processing in near real-time. Apache Kafka is a key technology used in Siphon, as its scalable pub/sub message queue. Siphon handles ingestion of over a trillion events per day across multiple business scenarios at Microsoft. Initially Siphon was engineered to run on Microsoft’s internal data center fabric. Over time, the service took advantage of Azure offerings such as Apache Kafka for HDInsight, to operate the service on Azure.

Here are a few of the scenarios that Siphon supports for Microsoft:

O365 Security: Protecting Office 365 customers’ data is a critical part of the business. A critical aspect of this is detecting security incidents in near real-time, so that threats can be responded to in a timely manner. For this, a streaming processing pipeline processes millions of events per second to identify threats. The key scenario requirements include:

◈ Ingestion pipeline that reliably supports multiple millions of events/second
◈ Reliable signal collection with integrated audit and alert
◈ Support O365 compliance certifications such as SOC and ISO

For this scenario, Siphon supports ingestion of more than 7 million events/sec at peak, with a volume over a gigabyte per second.

O365 SharePoint Online: To power analytics, product intelligence, as well as data-powered product features, the service requires a modern and scalable data pipeline for connecting user activity signals to the downstream services that consume these signals for various use cases for analytics, audit, and intelligent features. The key requirements include:

◈ Signals are needed in near real-time, with end to end latency of a few seconds
◈ Pipeline needs to scale to billions of events per day
◈ Support O365 compliance and data handling requirements

Siphon powers the data pub/sub for this pipeline and is ramping up in scale across multiple regions. Once the service was in production in one region, it was an easy task to replicate it in multiple regions across the globe.

MileIQ: MileIQ is an app that enables automated mileage tracking. On the MileIQ backend, there are multiple scenarios requiring scalable message pub/sub:

◈ Dispatching events between micro-services
◈ Data integration to the O365 Substrate
◈ ETL data for analytics

MileIQ is onboarding to Siphon to enable these scenarios which require near real-time pub/sub for 10s of thousands of messages/second, with guarantees on reliability, latency and data loss.

Siphon architecture

Siphon provides reliable, high-throughput, low-latency data ingestion capabilities, to power various streaming data processing pipelines. It functions as a reliable and compliant enterprise-scale ‘Data Bus.’ Data producers can publish data streams once, rather than to each downstream system; and data consumers can subscribe to data streams they need. Data can be consumed either via streaming platforms like Apache Spark Streaming, Apache Storm, and more, or through Siphon connectors that stream the data to a variety of destinations.

A simplified view of the Siphon architecture:

The core components of Siphon are the following:

◈ Siphon SDK: Data producers send data to Siphon using this SDK, that supports schematizing, serializing, batching, retrying and failover. 

◈ Collector: This is a service with an HTTPS endpoint for receiving the data. In provides authentication, routing, throttling, monitoring and load balancing/failover.

◈ Apache Kafka: One more Kafka clusters are deployed as needed for the scenario requirements.

Connectors: A service that supports config-driven movement of data from Siphon to various destinations, with support for filtering, data transformation, and adapting to the destination’s protocol.

These components are deployed in various Microsoft data centers / Azure regions to support business scenarios. The entire system is managed as a multi-user/multi-tenant service with a management layer including monitoring and alerting for system health, as well as an auditing system for data completeness and latency.

Siphon’s journey to HDInsight

When the Siphon team considered what building blocks they needed to run the service on Azure, the Apache Kafka for HDInsight service was an attractive component to build on. The key benefits are:

◈ Managed service: The HDInsight service takes care of Apache Kafka cluster creation, and keeping the clusters up and running, and routine maintenance and patching, with an overall SLA of 99.9 percent.

◈ Compliance: HDInsight meets a number of security and compliance requirements and is a good foundation from which Siphon could build additional capabilities needed to meet the stringent needs of services like Office 365.

◈ Cost: Innovations such as integration of the Kafka nodes with Azure Managed Disks enable increased scale and reduced cost without sacrificing reliability.

◈ Flexibility: HDInsight gives the flexibility to customize the cluster both in terms of the VM type and disks used, as well as installation of custom software, and tuning the overall service for the appropriate cost and performance requirements.

Siphon was an early internal customer for the Apache Kafka for HDInsight (preview) service. Implementation of the Azure Managed Disk integration enabled lowering the overall cost for running this large scale ‘Data Bus’ service.

Siphon currently has more than 30 HDInsight Kafka clusters (with around 600 Kafka brokers) deployed in Azure regions worldwide and continues to expand its footprint. Cluster sizes range from 3 to 50 brokers, with a typical cluster having 10 brokers, with 10 disks attached to each broker. In aggregate, these Siphon clusters support ingesting over 4 GB of data per second at peak volumes.

Apache Kafka for HDInsight made it easy for Siphon to expand to new geo regions to support O365 services, with automated deployments bringing down the time to add Siphon presence in a new Azure region to hours instead of days.

Continue reading

Exciting advances in Azure Alerts – From better alert management to Smart Groups

We are excited to announce preview of three new features in Azure Monitor that let you enumerate alerts at scale across log, metric or activity log alerts, filter alerts across subscriptions, manage alert states, look at alert instance specific details, and troubleshoot issues faster using SmartGroups that automatically group related alerts. These features continue to enhance the unified alerts configuration experience announced earlier this year. We look forward to your feedback to refine the functionality further.

The new alert enumeration experience and API allows observing alerts across Azure deployments. Alerts across multiple subscriptions can be queried and pivoted on severity, signal types, resource type, and more allowing a performant and easy summary-to-drill down experience. The new enumeration experience also supports multi-select filtering on any relevant dimension, allowing for example, looking up alerts across a set of resource groups or specific resource types.

Alert state management provides users a way to change the state of the alert to reflect the current situation of the issue in their environment. Currently three alert states are supported – New, Acknowledged, and Closed.

Alert states are separate from the monitoring condition, which is updated by the underlying monitoring service that detected the issue. Monitoring condition supports two values – fired and resolved.

The history of both monitor condition and alert state changes, as well as the details of the event such as target resource uri, alert rule, monitor condition, and link to query for log alerts, are captured in the payload of the alert to aid in triaging and auditing.

Smart Groups are system generated alerts that encapsulate many related alerts to reduce alert noise and help in mitigating events faster. These Smart Groups are automatically created using machine learning algorithms looking for similarity and co-occurrence patterns among alerts originating from a monitor service (e.g. LA or Platform). Smart Groups have the same properties as an individual alert such as user defined states or history. For an operator, smart groups significantly reduce the number of alerts to analyze by focusing on only a handful of groups. For example, if % CPU on several virtual machines in a subscription simultaneously spikes leading to many individual alerts, and if such alerts have occurred together anytime in the past, these alerts will likely be grouped into a single Smart Group, suggesting a potential common root cause. This technology has gone through extensive testing against hyperscale Azure services with very good results.

Continue reading

SmartHotel360 Microservices on Azure Kubernetes Service

During the Build 2018 keynote, Scott Hanselman, with help from Scott Guthrie, showed developers the new features available in Azure Kubernetes Service (AKS) and Azure Dev Spaces. To help you learn how to deploy microservices written in any framework to AKS we've updated the SmartHotel360 back-end microservices source code and deployment process to optimize it for AKS. You can clone, fork, or download the AKS and Azure Dev Spaces demo on GitHub.

Azure Kubernetes Service gives developers the best experience for building microservices in any platform including Java, .NET Core, or Node.js, to name a few used in this demo's source code, using Kubernetes and containers. The diagram below shows a high-level snapshot of the back-end APIs housed in the AKS cluster once you deploy this repository's source code to AKS.

Sample queries shipped with the demo can be saved and executed in an AKS cluster to get deep visibility into how each service is running. The query below, for example, results with a bar chart showing how often a certain pattern appears in the execution logs.

Azure Dev Spaces provides a rapid, iterative Kubernetes development experience for teams. With minimal dev machine setup, you can iteratively run and debug containers directly in Azure Kubernetes Service. Develop on Windows, Mac, or Linux using familiar tools like Visual Studio, Visual Studio Code, or the command line. The diagram below shows how the Visual Studio family of IDEs can connect to AKS to enable debugging within a developer’s Azure Dev Space without impacting production or teammate code.

The AKS Cluster created by the demo contains support for Azure Dev Spaces, so that you can debug the individual services live in the Kubernetes cluster. There's a pre-wired error in the hotels microservice you'll fix during the demo, then debug in your own Azure Dev Space to validate the fix worked.

Demo contents

The sample repository contains:

◈ The source code and deployment for the back-end services and the public web app.

◈ The demo script and the setup instructions, written in bash, so it can be easily executed on Linux, Mac, or in Windows using WSL.

◈ Helm charts for each service.

◈ Sample queries for use in with AKS Log Search.

◈ A preloader script that can be used to generate log/CPU data.

The SmartHotel360 public web site was originally written to demonstrate the features that make Azure App Service the best place in the cloud to host ASP.NET Core applications, with amazing diagnostics, deployment, and devops features.

AKS is a great place to host ASP.NET Core applications, too, so to give you great examples of both scenarios, we've moved the public web site into the AKS cluster for this sample. If you're investigating the variety of options for hosting your ASP.NET Core apps in Azure, you'll have the original App Service-focused version of the demo source code, and you'll learn from the new demo repository how to publish an ASP.NET Core app into AKS.

Azure Kubernetes Service brings so many amazing features for developers. The container health dashboard, deep log search features enabling you to really see how your code's executing in the cluster, and IDE/debugger integration that makes it possible for you to edit and debug code live in the cluster without impacting production or teammate code all make AKS the greatest experience for building apps with Kubernetes.

We hope this demo is useful in your process of learning how to publish microservices to AKS and to make the most use of the amazing portal and debugging features. As with all of the SmartHotel360 repositories, these are open and we encourage pull requests. If you experience any issues setting it up, send us an issue in GitHub and we'll resolve it quickly.

Continue reading

Soft delete for Azure Storage Blobs generally available

Today we are excited to announce general availability of soft delete for Azure Storage Blobs! The feature is available in all regions for public, government and sovereign clouds.

When turned on, soft delete enables you to save and recover your data where blobs or blob snapshots are deleted. This protection extends to blob data that is erased as the result of an overwrite.

How does it work?

When data is deleted, it transitions to a soft deleted state instead of being permanently erased. When soft delete is on and you overwrite data, a soft deleted snapshot is generated to save the state of the overwritten data. Soft deleted objects are invisible unless explicitly listed. You can configure the amount of time soft deleted data is recoverable before it is permanently expired.

Soft deleted data is grey, while active data is blue. More recently written data appears beneath older data. When B0 is overwritten with B1, a soft deleted snapshot of B0 is generated. When the blob is deleted, the root (B1) also moves into a soft deleted state.

Soft delete is 100 percent backwards compatible; you don’t have to make changes to your applications to take advantage of the protections this feature affords. With this GA announcement, we have added support for tiering blobs with soft deleted snapshots. When Set Blob Tier is called on a blob with soft deleted snapshots, the snapshots will remain in the original storage tier and expire based on the retention period you configured.

When you create a new account, soft delete is off by default. Soft delete is also off by default for existing storage accounts. You can toggle the feature on and off at any time during the life of a storage account. Object-level soft delete is available for all storage account types and all storage tiers. It does not protect against container or account deletions.

Soft deleted data is billed at the same rate as active data.

Getting started

Soft delete is supported by Azure Portal, .NET Client Library (version 9.0.0), Java Client Library (version 7.0.0), Python Client Library (version 1.1.0), Node.js Client Library (version 2.8.0), PowerShell (version 5.3.0) and CLI 2.0 (version 2.0.27). You can also directly use the Storage Services REST API as always. Soft delete is supported by REST API version 2017-07-29 and greater. In general, we always recommend using the latest version regardless of whether you are using this feature.

To enable soft delete using the Azure Portal, navigate to the "Soft delete" option under "Blob Service." Then, click "Enabled" and enter the number of days you want to retain soft deleted data.

If there is a chance that your data is accidentally modified or deleted by an application or other storage account user, we recommend turning on soft delete. Soft delete is one part of a data protection strategy and can help prevent inadvertent data loss.

Soft delete helps ensure that you can recover accidentally deleted or modified blob data. Soft delete is a key part of an overall data protection strategy that includes Azure Resource Manager locks as well as the ZRS, GRS, and RA-GRS replication tiers.

Continue reading

Azure Data Lake Tools for VSCode supports Azure blob storage integration

We are pleased to announce the integration of VSCode explorer with Azure blob storage. If you are a data scientist and want to explore the data in your Azure blob storage, please try the Data Lake Explorer blob storage integration. If you are a developer and want to access and manage your Azure blob storage files, please try the Data Lake Explorer blob storage integration. The Data Lake Explorer allows you easily navigate to your blob storage, access and manage your blob container, folder and files.

Summary of new features

◈ Blob container - Refresh, Delete Blob Container and Upload Blob

◈ Folder in blob - Refresh and Upload Blob 

◈ File in blob - Preview/Edit, Download, Delete, Create EXTRACT Script (only available for CSV, TSV and TXT files), as well as Copy Relative Path, and Copy Full Path

How to install or update

Install Visual Studio Code and download Mono 4.2.x (for Linux and Mac). Then get the latest Azure Data Lake Tools by going to the VSCode Extension repository or the VSCode Marketplace and searching Azure Data Lake Tools.

Continue reading