Wednesday, 21 November 2018

Simplifying confidential computing: Azure IoT Edge security with enclaves – Public preview

Enabling Open Enclave SDK for the intelligent edge and simplifying the development of trusted applications across operating systems and hardware platforms.

Today we are excited to announce the cross-platform availability of the Open Enclave SDK for devices at the intelligent edge, as well as the preview of integration of Open Enclave and the Azure IoT Edge security manager. Together with our partners, we help organizations solve a very challenging security problem in the Internet of Things (IoT) – protecting code and data while in use at the edge. This includes securing compute workloads within the confines of Trusted Execution Environments (TEE) or simply enclaves, a concept called Confidential Computing.  We offer a platform for developing applications that execute in enclaves, also known as trusted applications (TA), in a way that simplifies TA development for all audiences from hardcore hardware security experts to edge and cloud software applications developers.

With today’s announcement, Azure IoT Edge application developers can write TAs that root trust in any secure silicon TEE built on such enclaving technologies like ARM TrustZone®, Intel SGX, and embedded Secure Elements using Windows or Linux operating systems. This broad applicability across different enclave technologies and integration with IoT Edge greatly simplifies the work developers must do to protect data and devices at the edge. With accessibility by all security expertise as topmost goal, this integration is laden with features to truly simplify and shorten the journey from idea to at-scale production deployment of secure intelligent edge applications.

Allow me to provide some context.

The need for security at the intelligent edge


The intelligent edge extends cloud intelligence and analytics to edge devices. Azure IoT Edge is a fully managed service that delivers cloud intelligence locally by deploying Azure AI, Azure services, and custom logic directly on edge devices, such as gateway class devices. IoT Edge lets organizations bring AI and cloud analytics to the edge in cases where poor/no connectivity, high latency, or high costs would have prevented connection to the cloud.

Security is crucial for trust and integrity at the edge because by their very nature, IoT devices may not always be in trusted custody. More so, intelligent edge devices are much “smarter” than most IoT devices; in contrast to the common sensors and actuators class of IoT devices, they process highly sensitive workloads to generate equally sensitive results, thereby requiring strong protection for security goals like integrity, privacy, and confidentiality. Prior to the intelligent edge and edge-class IoT devices, these sensitive workloads existed only within the safety of the cloud. As these organizations move these sensitive workloads to the intelligent edge, they face increased risk from threats like tampering and forgery because the devices are often easily physically accessible. To reduce that risk, we created Azure Sphere, a solution for building highly-secured microcontroller devices, to help organizations build trustworthy devices for edge applications.

We recognize, however, that that IoT devices derive from a wide variety of hardware to allow for optimal solutions to the problems IoT targets; not all edge devices can be built on a microcontroller, meaning organizations need a greater variety of devices than they can build with Azure Sphere. With that variety, unfortunately, comes an expanded threat surface. For security to be effective, the solution must be uniform across devices to enable safe operation at scale. The heterogeneous nature of hardware in edge devices at the edge, more than its cloud counterpart, calls for very strong diligence. Just like the cloud, the intelligent edge requires a strong stance on security that addresses all threats, including emerging threats specific to operating at the edge.

With security in forethought, we designed Azure IoT Edge with a solid foundation of security principles for the intelligent edge.  One year ago together with partners, we proved these principles thereby paving the way for all the possibilities towards achieving the highest security at the edge, but we did not stop there. We needed to scale this model by tapping into the domain expertise of the manufacturers who produce hardware root of trust modules. We also needed to isolate the IoT and cloud application developer from the complexity and nuances of working with secure silicon hardware. We achieved both goals with Azure IoT Edge security manager, a well-bounded trusted computing base whose sole mission is to protect the Azure IoT Edge device and its components by rooting the identity and sensitive workloads of the device in secure silicon also commonly known as a hardware security module (HSM). Azure IoT Edge security manager spurred the availability of secured edge devices from various Original Equipment Manufacturers (OEMs) as standard commercial off-the-shelf offerings. Although we’re happy with more availability of secured edge devices for our customers, we’re not done with our plan to increase the security of the intelligent edge.

Our next mission for securing the intelligent edge is to simplify access to security best practices for experts and non-experts for all security challenges, especially the big challenges. Until now, the ability to secure data at rest and data in transit has remained the sole province of security experts; on the other hand, it is generally considered a “solved problem,” meaning that non-experts can take advantage of the work experts have done and technology vendors have adopted. Emerging with the intelligent edge, however, is the need to protect data in use at the edge, and this is a tall challenge even to the experts. Confidential computing at the edge is today’s immediate, big security challenge. The solution is never to lower the bar for security, but to rise above the challenge and empower every application developer, security experts and non-experts alike, with the right tools.

Two months ago, Microsoft launched the Open Enclave SDK, an open source consistent API surface across enclave technologies and platforms from cloud to edge. This means that developers can use the same APIs across multiple enclaves, greatly reducing the complexity of following best practices and encouraging organizations to integrate applications with enclaves. Available with this version was the ability to write enclave applications for cloud workloads targeting TEE technology based on Intel Software Extension Guard (SGX) hardware technology and the Linux Operating System.

Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Figure 1. Architecture of cloud-edge consistent confidential computing framework showing where the Open Enclave API and abstraction fits in intelligent edge scenarios based on Azure IoT Edge. Open Enclave API offers the interface between TEE and a non-TEE environment commonly known as the Rich Execution Environment (REE). Underneath the Open Enclave API, adopters have the option of choosing Open Enclave SDK or an equivalent implementation from a third party offering.

Features


Secure silicon hardware abstraction

Open Enclave abstracts the developer experience away from the complexity and nuances of secure hardware, meaning that its easier for developers to learn to integrate their apps with secure hardware and easier for developers to write consistently secure software across devices.

Tooled for familiarity

Support for Visual Studio for Windows and GDB for Linux is available now so that developers can engage in already familiar development environments of choice.

Trusted Execution Environment (TEE) emulation for richer Trusted Application (TA) development

Until now, development of enclave applications or TA, especially for edge scenarios, has been the sole domain of a few experts with extensive experience in the specific TEE hardware offering. Development required access to the hardware, and debugging was extremely challenging given enclaving is about protecting data. We’re changing that by providing TEE emulation so that any developer, not just TEE hardware experts, can develop and debug trusted applications using familiar tools and with no need for the hardware. Developers can develop once and compile for multiple TEE hardware technology targets.

Designed for cloud-edge consistent confidential computing

Confidential compute resources are different for cloud and edge but the development experience for confidential compute workloads targeting the cloud and edge need not be different. In a single development environment, developers can create trusted applications for deployment in cloud enclaves like Azure Confidential Computing and in TEE enabled IoT Edge devices. Developers can create holistic security for applications where code and data are protected at rest, in flight, and while in use at the edge and in the cloud. In the very near future, they’ll be able to leverage Azure Device Provisioning Service (DPS), an at scale global provision service to provision and manage the lifecycle of confidential compute workloads and trusted applications consistently across cloud and edge.

Rich and diverse partner ecosystem

We strongly believe security in IoT is a community play. IoT deployments require trust from the device manufacturer to the solution integrator and solution backend—for this reason we’re aligning with industry leaders in various roles to enable truly end-to-end security for IoT devices, connections, and cloud.

Technology Silicon provider   Original design manufacturer (ODM)  Original equipment manufacturer (OEM)  Independent software vendor (ISV)  Mass production & distribution 
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things

Application possibilities with confidential computing using Azure IoT Edge


Azure IoT Edge with Open Enclave uses confidential computing to provide solutions to security challenges previously very difficult to surmount. It really comes down to protecting the executable delivering the business function.

Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things

TrustBox – One optimal path from evaluation to mass production


As organizations learn about the intelligent edge and begin projects, they often find themselves slowed down by unforeseen security and compliance challenges. In response, we are working with partners to shorten the journey from evaluation of a project to production. Until now, a typical experience entails developers evaluating and experimenting using many commercially available prototyping boards, such as Arduino-based boards or Raspberry Pis. Unfortunately, when the developers move to production, they find they have to switch boards to a solution that can meet the required security and compliance standards for their industry. Rather than switching boards later, developers should begin evaluation with boards that can later serve pilot and production needs while meeting compliance goals.

TrustBox is one option to address this need for prototype to production boards, providing a high-grade secure router and IoT gateway based around NXP Layerscape LS1012A networking processor that integrates hardware root of trust, cryptographic accelerators and network acceleration. Following deep security engineering and careful selection of components, TrustBox is certifiable to the highest levels of industry and security standards. TrustBox is the winner of the award for 2019 CES Best of Innovations: Cybersecurity and Personal Privacy.

Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Figure 2: TrustBox, a high-grade secure router & IoT gateway based on NXP Layerscape LS1012A, a member of NXP’s Layerscape family of hardware root of trust and cryptographic accelerators. TrustBox is available as is or customizable through Scalys for evaluation, pilot and production runs.

Availability of evaluation boards is just one of many steps to mass production which can be daunting where there is need to customize the boards. To this end we allied with NXP, Scalys, and Arrow Electronics, to provide an example of a complete supply chain from integrated circuit to final distribution.

Azure Certification, Azure Tutorial and Material, Cloud Strategy, Internet of Things
Figure 3: The supply chain for TrustBox that is available now for all enclave secured device needs to include sourcing for evaluation and pilot runs, hardware customizations, and mass production. This is only one of many possibilities.

Scalys TrustBox is expected to ship in volume from February 2019, and you can pre-order from Scalys now. You can also start evaluating now with Grapeboard, the commercial grade electrical equivalent of TrustBox.

Saturday, 17 November 2018

Azure Toolkit for IntelliJ – Spark Interactive Console

We are pleased to reveal the release of Spark Interactive Console in Azure Toolkit for IntelliJ. This new component facilitates Spark job authoring, and enables you to run code interactively in a shell-like environment within IntelliJ.

The Spark console includes Spark Local Console and Spark Livy Interactive Session. When you run the Spark console, instances of SparkSession and SparkContext are automatically instantiated like in Spark shell. You can use 'spark' to access the SparkSession and use 'sc' to access the SparkContext. The Spark local console allows you to run your code interactively and validate your code logic locally. You can also check your programming variables and perform other scripting operations locally before submitting to the cluster. The Spark Livy interactive session establishes an interactive communication channel with your cluster so you can check on file schemas, preview data, and run ad-hoc queries while you are programming your Spark job. You can also easily switch the Livy interactive session against different Spark clusters.

The Spark console has a Language Service built-in for Scala programming. You can leverage the language service features, such as IntelliSense and autocomplete, to look up a Spark object (i.e., Spark context and Spark session) properties, query hive metadata, and check on function signatures.

A new feature, Send Selection to Spark Console (Ctrl + Shift  + S), has been added to simplify the user experience for accessing the Spark console. You can send a highlighted single line of code or a block of code to the console from your main Scala project. This feature enables you to switch smoothly between contexts: coding and validation or testing code in the Spark console.

Summary of new features


◈ Run Spark local console
◈ Run Spark Livy interactive session console
◈ Language service for Scala enabled in the console
◈ Send selected code to console

The addition of the Spark console is an important step forward for the Azure Toolkit because of its expanding capabilities beyond batch job processing. This update also supports interactive querying across local and dev/test clusters.

To run your code and, please select and hold Ctrl + Enter, and use the up and down arrows to browse the history of previously run code.

Azure Certification, Azure Study Materials, Azure Tutorial and Materials, Azure Learning

Azure Certification, Azure Study Materials, Azure Tutorial and Materials, Azure Learning

How to access


You can easily start the Spark console either from the Tools menu or from the Scala file by right-clicking on the context menu.

Azure Certification, Azure Study Materials, Azure Tutorial and Materials, Azure Learning

Azure Certification, Azure Study Materials, Azure Tutorial and Materials, Azure Learning

Friday, 16 November 2018

Introducing the Azure Blockchain Development Kit

“Developers! Developers! Developers!” That phrase is synonymous with Microsoft’s history of democratizing complex technologies and empowering anyone with an idea to build software.

Over four decades, we’ve lowered barriers to development with developer tooling, enterprise integration, DevOps, PaaS, and SaaS. Today, serverless offerings from Functions and Logic Apps to Azure DevOps and IoT Central remove friction for development in the cloud.

This morning, we’re excited to announce the initial release of the Azure Blockchain Development Kit which is built on Microsoft’s serverless technologies and seamlessly integrates blockchain with the best of Microsoft and third-party SaaS.

This kit extends the capabilities of our blockchain developer templates and Azure Blockchain Workbench, which incorporates Azure services for key management, off-chain identity and data, monitoring, and messaging APIs into a reference architecture that can be used to rapidly build blockchain-based applications.

These tools have become the first step for many organizations on their journey to re-invent the way they do business. Apps have been built for everything from democratizing supply chain financing in Nigeria to securing the food supply in the UK, but as patterns emerged across use cases, our teams identified new ways for Microsoft to help developers go farther, faster.

This initial release prioritizes capabilities related to three key themes: connecting interfaces, integrating data and systems, and deploying smart contracts and blockchain networks.

Connect


To deliver end to end blockchain solutions for consortiums, developers need to enable organizations, people, and devices to connect to the blockchain and do it from a heterogenous set of user interfaces.

Take for example an end to end supply chain for a commodity such as cocoa.

◈ SMS and voice interfaces enable small hold farmers in Africa to transact and track their goods at the first mile of the supply chain.
◈ Internet of Things (IoT) devices deliver sensor data to track the conditions of the goods at different points in their journey to market – tracking the humidity in the containers where the beans are held to the temperature of the end product of ice cream that it is incorporated into.
◈ Mobile clients enable logistics providers to accept and transfer responsibility for products on their journey from manufacturer to retail using the compute power that already exists in the pockets of its employees. Mobile devices also have sensors such as GPS and cameras that can add complementary data that can help attest to the what, where, and when of deliveries.
◈ Backend Systems and Data in the form of ERP systems such as Dynamics and SAP are used to manage core processes for different participants. These systems also become clients via extension and need to interact with smart contracts to provide and receive attestable data on behalf of an organization.
◈ Bots and assistants enable manufacturers and retailers to interact with the supply chain. This includes interacting with smart contracts for orders and provenance using natural language and using attestable data from the blockchain to direct actions taken on behalf of a user.
◈ Web clients enable end consumers to query the origin of the product purchased at retail, typically a mix of provenance and story of their journey of their product from “farm to fork”
The Azure Blockchain Development Kit includes samples for all of these scenarios, including inbound and outbound SMS, IVR, IoT Hub and IoT Central, Xamarin mobile client for iOS and Android, Dynamics integration via Common Data Service (CDS), bots and assistants (Cortana, Alexa, Google Assistant) and web UX.

Integrate


Businesses are using blockchain and smart contracts to facilitate multi-party processes. Blockchain also delivers real-time transparency of the states and events of those contracts to appropriate participants.

End to end blockchain solutions require integration with data, software, and media that live “off chain”. External updates and events can trigger actions on smart contracts. Smart contract events and state changes can then trigger actions and data updates to “off chain” systems and data. These external systems and AI will also need the ability to query attestable data from smart contracts to inform action.

Specifically, there are two areas of integration where guidance is most needed:

Documents and Media: Documents and media do not belong on chain, but business processes often involve images, videos, audio, Office documents, CAD files for 3D printers or other file types.

The common pattern is to generate a unique hash of the media and the metadata that describes it. Those hashes are then placed on a public or private chain. If authenticity of a file is ever questioned, the “off chain” files can be re-hashed at a later time and that hash is compared to the “on chain” hash stored on the blockchain. If the hashes match, the document is authentic, but if so much as a pixel of an image or letter in a document is changed, the hashes will not match and this will make obvious that tampering has occurred.

Today we’re releasing a set of Logic Apps that enable the hashing of files and file related metadata. Also included are smart contracts for files and a file registry to store the hashes on chain.

Logic Apps have been created to deliver this functionality for files added to the most popular sources for documents and media, including Azure Storage, OneDrive, One Drive for Business, SharePoint, Box, Adobe Creative Cloud, and FTP.

Azure Blockchain Development Kit, Azure Certification, Azure Tutorial and Materials, Azure Learning

Smart Contract Interaction: Getting blockchain off the whiteboard and into production means dealing with the realities of how counterparties interact today. That reality is that Enterprise integration is messy.

Microsoft brings our decades of experience in this area to blockchain. Our work with integrating Enterprise systems began almost two decades ago with the introduction of BizTalk server, and our focus on database integration traces back to our co-development of Open Database Connectivity (ODBC) in the 1990s. All of our experience has been captured and made available in Azure services. This includes 200+ connectors available in Logic Apps and Flow, and the robust capabilities in our data platform.

Azure Blockchain Development Kit, Azure Certification, Azure Tutorial and Materials, Azure Learning

The Blockchain Application Development Kit includes Workbench integration samples in the following areas:

◈ Legacy applications and protocols – Sending and receiving files via FTP, processing comma separated files and email delivery of data.
◈ Data – SQL, Azure Search, Excel, and PowerBI.
◈ SaaS –  SharePoint, Dynamics, Outlook, and Gmail.
◈ Registries – An accelerator that generates a custom registry and registry item smart contracts to accommodate any scenario.

Logic App Connectors for Blockchain


Today, we are also announcing that we will release a set of Logic App and Flow Connectors to extend these samples to ledgers like Ethereum, Corda, Bitcoin, and others.

"At R3, we are committed to ensuring developers can deploy CorDapps quickly, securely and easily. The Azure Blockchain Development Kit will give our enterprise customers tools to integrate with the applications, software, and devices that people use every day like Outlook, Alexa, SMS, and web UX. Blockchain is moving out of the labs and into everyday business applications.”

– Mike Ward, Head of Product Management, R3

The Ethereum blockchain connector is available today and enables users to deploy contracts, call contract actions, read contract state and trigger other Logic Apps based on events from the ledger.

Azure Blockchain Development Kit, Azure Certification, Azure Tutorial and Materials, Azure Learning

Deploy

With the mainstreaming of blockchain technology in enterprise software development, organizations are asking for guidance on how to deliver DevOps for smart contracts and blockchain projects.

Common questions include:

◈ My business logic and data schema for that logic is reflected in smart contracts. Smart contracts are written in languages I’m less familiar with like Solidity for Ethereum, and Kotlin for Corda, or Go for Hyperledger Fabric.  What tools can I use to develop those in?

◈ How do I do unit testing and debugging on smart contracts?

◈ Many blockchain scenarios reflect multi-party transactions and business workflows. These workflows include signed transactions from multiple parties happening in specific sequences. How do I think about data for test environments in that context?

◈ Smart contracts are deployed to the blockchain, which is immutable. How do I need to think about things such as infrastructure as code, local dev/test, upgrading contracts, etc.?

◈ Blockchain is a data technology shared across multiple organizations in a consortium, what are the impacts on source code control, build and release pipelines in a global, multi-party environment?

While there are some nuances to the approach, the good news is that just like other types of solution development, this model can readily be addressed in a DevOps model.

Azure Blockchain Development Kit, Azure Certification, Azure Tutorial and Materials, Azure Learning

“We're excited to work with Microsoft to create the canonical DevOps experience for blockchain engineers. Our paper, ‘DevOps for Blockchain Smart Contracts’, goes into rigorous detail and provides examples on how to develop blockchain applications with an eye toward CI/CD in consortium environments.”

- Tim Coulter, Founder of Truffle

Complementing the whitepaper is an implementation guide, available through the Azure Blockchain Development Kit, that shows how to implement CI/CD for smart contracts and infrastructure as code using Visual Studio Code, GitHub, Azure DevOps and OSS from Truffle.

A great platform for blockchain application development

The Azure Blockchain Development Kit is the next step in our journey to make developing end to end blockchain applications accessible, fast, and affordable to anyone with an idea. It is built atop our investments in blockchain and connects to the compute, data, messaging, and integration services available in both Azure and the broader Microsoft Cloud to provide a robust palette for a developer to realize their vision.

Logic Apps and Flow deliver a graphical design environment with more than 200 connectors dramatically simplifying the development of end to end blockchain solutions, and Azure Functions enable the rapid integration of custom code.

A serverless approach also reduces costs and management overhead. With no VMs to manage, built-in scalability, and an approachable pricing model the Azure Blockchain Development Kit is within reach of every developer – from enthusiasts to ISVs to enterprises.

Solutions are written using online visual workflow designers and Visual Studio Code, a free download that provides an integrated development environment on Windows, Mac, and Linux.

The resulting applications will run atop a network that has higher rated cloud performance than other large-scale providers and enable federating identities between participants using Azure Active Directory. With Azure, those applications can be deployed to more regions than any other cloud provider and benefit from more certifications.

We look forward to seeing what you’ll build, and we’ll continue to both listen and look for ways to help as we build a decentralized future together.

Get started in minutes: Build your own enterprise grade virtual assistant

Whether in a vehicle looking to find the nearest gas station, or on a factory floor needing to stop the production line, it is easy to imagine ways in which conversation-based interactions can enhance experiences and increase productivity. We’re evolving from a world where humans have had to learn and adapt to computers to one where computers are learning how to understand and interact with humans in a more natural way.

Azure Bot Service helps developers build, connect, deploy, and manage intelligent bots to interact naturally with users on websites, apps, Cortana, Microsoft Teams, Skype, Slack, Facebook Messenger, and more. With over 360,000+ registered Azure Bot Service developers, we are constantly improving our tools and framework to make them more productive. Today, we’re announcing the preview of an open source bot solution accelerator for virtual assistants. This solution accelerator simplifies the creation of branded virtual assistants, enabling developers to get started in minutes. Additionally, the Bot Framework software development kit and Tools version 4.1 is generally available, enabling developers to be more productive with bot development. Finally, Microsoft has signed an agreement to acquire XOXCO, a software product design studio most known for its conversational AI and bot development capabilities.

Create a branded virtual assistant


We have seen a significant need from our customers and partners to deliver a conversational assistant tailored to their brand, personalized to their customers, and made available across a broad range of conversational applications and devices.

We believe all organizations should have the ability to build branded virtual assistants. We have taken best practices developed while working with partners to develop their own assistants and, in continuing with Microsoft’s open source approach for bot development, have made this solution accelerator for virtual assistants available on GitHub, allowing for full control over the end user experience built on a set of foundational capabilities. Additionally, the experience can be infused with intelligence about the end-user and any device/ecosystem for a truly integrated conversational experience.

The scope of this functionality is broad, typically offering end users a range of capabilities. To increase developer productivity and enable a vibrant ecosystem of reusable conversational experiences, we are providing developers initial examples of reusable conversational skills. These skills can be added into a conversational application to light up a specific conversation experience, such as finding a point of interest, interacting with a calendar, tasks, or email, with more to follow. Skills are fully customizable and consist of language models for multiple languages, dialogs, and code.

Azure Study Materials, Azure Guides, Azure Certification, Azure Learning

Bot Framework SDK and Tools Release 4.1


Following the successful general availability GA release of the Bot Framework SDK V4.0 during Microsoft Ignite 2018, the team is announcing further updates, including general availability (GA) of the Bot Framework Emulator, Web Chat control, and version 4.1 of the C# and JavaScript SDKs. 

The Bot Framework Emulator V4 simplifies integration and management of the different services that are part of a conversational experience. Developers can directly log into Azure and open and manage Cognitive Services commonly used to create conversational experiences such as: Language Understanding (LUIS), and QnA Maker.

The Web Chat control offers a quick way for developers to integrate their bots into existing websites. Web Chat V4 offers developers ability to fully customize the chat window, including ability to change colors, padding, sizes, and to brand the window with custom graphics and logos.

Azure Study Materials, Azure Guides, Azure Certification, Azure Learning

Microsoft to acquire XOXCO


We are also excited to share that Microsoft has signed an agreement to acquire XOXCO, a software product design and development studio known for its conversational AI and bot development capabilities. The company has been paving the way in conversational AI since 2013 and was responsible for the creation of Howdy, the first commercially available bot for Slack that helps schedule meetings, and Botkit, which provides the development tools used by hundreds of thousands of developers on GitHub. Over the years, we have partnered with XOXCO and been inspired by this work.

Azure Study Materials, Azure Guides, Azure Certification, Azure Learning

With this acquisition, we are continuing to realize our approach of democratizing AI development, conversation and dialog and integrate conversational experiences where people communicate.  We’re excited to welcome the XOXCO team and look forward to working with the community to accelerate innovation and help customers capitalize on the many benefits AI can offer.

Tuesday, 13 November 2018

Let AI help you be more productive with Microsoft Azure CLI

Keeping up with the pace of change in Microsoft Azure can be challenging. Every week there are more than 50 pull requests against the Azure Representational State Transfer (REST) API. Over the past 6 months, we’ve been building the AI-powered extension, Azure Aladdin, to help make using Azure easier. The first interface was a Microsoft docs extension that provided users content recommendations.

Today we are excited to offer the next interface to the Aladdin knowledge base, an experimental Azure Command-line interface (CLI) extension that provides insight and examples based on how other user have seen success using Microsoft Azure.

Install the Azure CLI "Find" extension


We’ve made it straightforward to install experimental extensions in Azure CLI. If you want to install the new “Find” extension, run the following command:

Microsoft Azure CLI, Azure Certification, Azure Guides, Azure Certification

For users of the Azure Cloud Shell, any extension installed will persist between sessions. Below we’re going to examine some of the capabilities of this new extension.

Explains CLI commands


For some features and previews, such as Azure Web App for Containers, there may only be auto-generated reference content without examples.

Microsoft Azure CLI, Azure Certification, Azure Guides, Azure Certification

Shows common commands in a group


The extension can also break down complex groups, such as Azure Monitor (az monitor), and provide the most common commands used:

Microsoft Azure CLI, Azure Certification, Azure Guides, Azure Certification

Help find commands


With more than 2,300 commands, it can be hard to find the right one. This extension replaces the existing Azure CLI "Find" command search capability with one powered by the knowledge base. Today, we provide a simple text search ordered by popularity with an example for each result but expect to see improvements over time.

Microsoft Azure CLI, Azure Certification, Azure Guides, Azure Certification

Examples, always up-to-date


When building tools for software development, reference documentation continues to be one of the most challenging areas to get right. Good reference documentation needs to be simple enough to allow beginners access, yet technical enough to enable professionals to support niche scenarios. Having up-to-date examples is key to providing good reference documentation.

By using the knowledge base and applying machine learning to generate examples, we’re able to ensure quality examples across our Azure management products and documentation. For example, creating a VM with "az vm create" offers 53 unique parameters used in hundreds of different ways. 

Today, only 42 percent of CLI commands have examples. Over the coming months, we will be integrating Aladdin-generated examples into the Azure CLI documentation, with a goal to reach 95 percent of all commands that come with usage examples.

How does this work?


To serve examples and related documentation, our robot is consulting a new knowledge base, called the Aladdin Knowledge Base. Aladdin is built by compiling information from many sources including our own documentation, code, and usage data, as well as Azure-related GitHub issues. This knowledge base is a continuation of our previous experiment, the Aladdin Doc Helper Chrome extension.

Microsoft Azure CLI, Azure Certification, Azure Guides, Azure Certification

“az find” consults this knowledge base to create the most relevant examples for each CLI command. If you’re one of the first few to learn a new command or service, you might not know it, but you are contributing by adding new entries to the knowledge base and helping make others successful. Each time Azure is used in a new way, our system incorporates that usage, along with docs.microsoft.com content to generate example ‘guide-posts’ to help future users along the way. It tracks the success of users who view our examples, a topic worthy of its own post and using the results to fine tune examples and recommendations. Our knowledge base is agnostic to any specific tool. In the future, we intend to create a  similar help module for Azure PowerShell that utilizes the same knowledge base.

This service is built entirely on Azure resources! Behind the scenes, we employ an Azure Search Service customized for our retrieval task and utilize Language Understanding Intelligent Service (LUIS) to allow our robots to understand humans.

Help us improve with your feedback


Help us make the tools that you love and use every day even better. How can you help? Provide feedback. With each Find result, you can provide feedback to let us know when Aladdin is useful, has lost its way or is giving bad advice. These reports are used to identify incorrect or out-of-date examples. Our robot will incorporate your feedback and will improve the quality and accuracy of examples, especially those related to uncommon scenarios.

Microsoft Azure CLI, Azure Certification, Azure Guides, Azure Certification

What’s next?


Our goal is to integrate this extension into the Azure CLI as a core feature. However, we’re also investigating some additional features:

◈ Personalization - Customize examples based on your profile, resources, and defaults.

◈ Keep up to date - The cloud changes fast, we’ll help you keep aware of services and features that are important to you.

◈ Migration information - When you are ready to migrate to a new feature or service, we will make sure you know your options, based on others who have previously migrated to Azure.

◈ Try this next - As you get more comfortable with the CLI, we’ll help you scale your solutions by showing you how others have scaled from where you are now.

We will continue to improve the Azure Aladdin knowledge base and offer more experiences and tools for managing Azure.

Monday, 12 November 2018

Static Data Masking for Azure SQL Database and SQL Server

The SQL Security team is pleased to share the public preview release of Static Data Masking. Static Data Masking is a data protection feature that helps users sanitize sensitive data in a copy of their SQL databases.

Azure SQL Database, SQL Server, Azure Guides, Azure Certification, Azure Learning

Use cases


Static Data Masking is designed to help organizations create a sanitized copy of their databases where all sensitive information has been altered in a way that makes the copy sharable with non-production users. Static Data Masking can be used for:

◈ Development and testing
◈ Analytics and business reporting
◈ Troubleshooting
◈ Sharing the database with a consultant, a research team, or any third-party

Static Data Masking facilitates compliance with security requirements such as the separation between production and dev/test environments. For organizations subject to GDPR, the feature is a convenient tool to remove all personal information while preserving the structure of the database for further processing.

How Static Data Masking works


With Static Data Masking, the user configures how masking operates for each column selected inside the database. Static Data Masking will then replace data in the database copy with new, masked data generated according to that configuration. Original data cannot be unmasked from the masked copy. Static Data Masking performs an irreversible operation.

In the example below, all entries in the column FirstName have been nullified. The column LastName is made of randomly generated strings. In the EmailAddress column, names have been replaced with randomly generated strings, but the domain extension has been maintained. A similar narrative applies to the Phone column where the area code has been preserved, but not the last 7 digits.

Azure SQL Database, SQL Server, Azure Guides, Azure Certification, Azure Learning

Static Data Masking vs. Dynamic Data Masking


Data masking is the process of applying a mask on a database to hide sensitive information and replace it with new data or scrubbed data. Microsoft offers two masking options, Static Data Masking and Dynamic Data Masking

Static Data Masking
Dynamic Data Masking
  • Happens on a copy of the database
  • Original data not retrievable
  • Mask occurs at the storage level
  • All users have access to the same masked data
  • Happens on the original database
  • Original data intact
  • Mask occurs on-the-fly at query time
  • Mask varies based on user permission

How to download Static Data Masking


Static Data Masking ships with SQL Server Management Studio 18.0. The latest preview SQL Server Management Studio 18.0 is available today for download.

Compatibility


Static Data Masking is compatible with SQL Server (SQL Server 2012 and newer), Azure SQL Database (DTU and vCore-based hosting options, excluding Hyperscale), and SQL Server on Azure Virtual Machines.

Sunday, 11 November 2018

Automating SAP deployments in Microsoft Azure using Terraform and Ansible

Deploying complex SAP landscapes into a public cloud is not an easy task. While SAP basis teams tend to be very familiar with the traditional tasks of installing and configuring SAP systems on-premise, additional domain knowledge is often required to design, build, and test cloud deployments.

There are several options to take the guesswork out of tedious and error-prone SAP deployment projects into a public cloud:

◈ One way to get started is the SAP Cloud Appliance Library (CAL), a repository of numerous SAP solutions that can be directly deployed into a public cloud. However, apart from its cost, CAL only contains pre-configured virtual machine (VM) images, so configuration changes are hard or impossible.
◈ A free alternative has been to use SAP Quickstart Templates offered by most public cloud providers. Typically written in a shell script or a proprietary language, these templates offer some customization options for pre-defined SAP scenarios. For example, Azure’s ARM templates offer one-click deployments of SAP HANA and other solutions directly in Azure Portal.)
While both solutions are great starting points, they usually lack configuration options and flexibility required to build up an actual, production-ready SAP landscape.

Based on feedback from actual customers who move their SAP landscapes into the cloud, the truth is that existing Quickstart Templates rarely go beyond “playground” systems or proof-of-concepts; they are too rigid and offer too little flexibility to map real-life business and technical requirements.

This is why we, the SAP on Microsoft Azure Engineering team, decided to go into the opposite direction: Instead of offering “one-size-fits-all” templates for limited SAP scenarios that can hardly be adapted (let alone extended), we broke down SAP deployments in Azure to the most granular level and offer “building blocks” for a truly customizable, yet easy-to-use experience.

A new approach to automating SAP deployments in the cloud


In this new, modular approach to automating even more complex SAP deployments in Azure, we developed a coherent collection of:

◈ Terraform modules which deploy the infrastructure components (such as VMs, network, storage) in Azure and then call the:
◈ Ansible playbook which call different:
◈ Ansible roles to install and configure OS and SAP applications on the deployed infrastructure in Azure.

Azure Tutorial and Material, Azure Guides, Azure Certification, Azure Study Materials

Flow diagram of Terraform/Ansible SAP automation templates.

An important design consideration was to keep all components as open and flexible as possible; although nearly every parameter on both Azure and SAP side can be customized, most are optional. In other words, you can be spinning up your first SAP deployment in Azure within 10 minutes by using one of our boilerplate configuration templates – but you can also use our modules and roles to build up a much more complex landscape.

Azure Tutorial and Material, Azure Guides, Azure Certification, Azure Study Materials

A sample deployment of HANA high-availability pair.

For your convenience, Terraform and Ansible are pre-installed in your Azure Cloud Shell, so the templates can be run directly from there with minimal configuration. Alternatively, you can, of course, use them from your local machine or any VM as well.

While the repository is published and maintained by Microsoft Azure, the project is community-driven and we welcome any contributions and feedback.

Starting with SAP HANA, but a lot more to come


When we started building our Terraform and Ansible templates a few months ago, we decided to start out our engineering process with HANA. SAP’s flagship in-memory database is the underlying platform and de-facto standard of most modern SAP enterprise applications, including S/4HANA and BW/4HANA. If you’ve ever built an SAP HANA high-availability cluster from scratch, you’ll appreciate that we’ve taken the guesswork out of this complex task and aligned our templates to the public cloud reference architectures certified by SAP.

Currently, our Terraform/Ansible templates support the following two options (more application-specific scenarios are currently being worked on):

HANA single-node instance


◈ Single-node HANA instance.

Azure Tutorial and Material, Azure Guides, Azure Certification, Azure Study Materials

HANA high-availability pair


◈ Single-node HANA instance, two-tier replication (primary/secondary) via HSR.
◈ Pacemaker high-availability cluster, fully configured with SBD and SAP/Azure resource agents.

Azure Tutorial and Material, Azure Guides, Azure Certification, Azure Study Materials

Since our key focus was to offer the greatest amount of flexibility possible, virtually every aspect of the SAP HANA landscape can be customized, including:

◈ Sizing (choose any supported Azure VM SKU).
◈ High-availability (in the high-availability pair scenario, choose to use availability sets or availability zones).
◈ Bastion host (optionally, choose from a Windows and/or Linux “jump box” including HANA Studio).
◈ Version (currently, HANA 1.0 SPS12 and HANA 2.0 SPS2 or higher are supported).
◈ XSA applications (optionally, enable XSA application server and choose from a set of supported applications like HANA Cockpit or SHINE).

Azure Tutorial and Material, Azure Guides, Azure Certification, Azure Study Materials

XSA SHINE demo content for HANA.

It’s worth noting that all scenarios come with “fill-in-the-blanks” boilerplate configuration templates and step-by-step instructions to help you get started.