Manage your big data needs with HDInsight on AKS

As companies today look to do more with data, take full advantage of the cloud, and vault into the age of AI, they’re looking for services that process data at scale, reliably, and efficiently. Today, we’re excited to announce the upcoming public preview of HDInsight on Azure Kubernetes Service (AKS), our cloud-native, open-source big data service, completely rearchitected on Azure Kubernetes Service infrastructure with two new workloads and numerous improvements across the stack.

HDInsight on AKS amplifying performance

HDInsight on AKS includes Apache Spark, Apache Flink, and Trino workloads on an Azure Kubernetes Service infrastructure, and features deep integration with popular Azure analytics services like Power BI, Azure Data Factory, and Azure Monitor, while leveraging Azure managed services for Prometheus and Grafana for monitoring. HDInsight on AKS is an end-to-end, open-source analytics solution that is easy to deploy and cost-effective to operate. 

HDInsight on AKS helps customers leverage open-source software for their analytics needs by: 

• Providing a curated set of open-source analytics workloads like Apache Spark, Apache Flink, and Trino. These workloads are the best-in-class open-source software for data engineering, machine learning, streaming, and querying.
• Delivering managed infrastructure, security, and monitoring so that teams can spend their time building innovative applications without needing to worry about the other components of their stack. Teams can be confident that HDInsight helps keep their data safe. 
• Offering flexibility that teams need to extend capabilities by tapping into today’s rich, open-source ecosystem for reusable libraries, and customizing applications through script actions.

Customers who are deeply invested in open-source analytics can use HDInsight on AKS to reduce costs by setting up fully functional, end-to-end analytics systems in minutes, leveraging ready-made integrations, built-in security, and reliable infrastructure. Our investments in performance improvements and features like autoscale enable customers to run their analytics workloads at optimal cost. HDInsight on AKS comes with a very simple and consistent pricing structure per vcore per hour regardless of the size of the resource or the region, plus the cost of resources provisioned.

Developers love HDInsight for the flexibility it offers to extend the base capabilities of open-source workloads through script actions and library management. HDInsight on AKS has an intuitive portal experience for managing libraries and monitoring resources. Developers have the flexibility to use a Software Development Kit(SDK), Azure Resource Manager (ARM) templates, or the portal experience based on their preference.

Open, managed, and flexible

HDInsight on AKS covers the full gamut of enterprise analytics needs spanning streaming, query processing, batch, and machine learning jobs with unified visualization. 

Curated open-source workloads

HDInsight on AKS includes workloads chosen based on their usage in typical analytics scenarios, community adoption, stability, security, and ecosystem support. This ensures that customers don’t need to grapple with the complexity of choice on account of myriad offerings with overlapping capabilities and inconsistent interoperability.  

Each of the workloads on HDInsight on AKS is the best-in-class for the analytics scenarios it supports: 

• Apache Flink is the open-source distributed stream processing framework that powers stateful stream processing and enables real-time analytics scenarios. 
• Trino is the federated query engine that is highly performant and scalable, addressing ad-hoc querying across a variety of data sources, both structured and unstructured.  
• Apache Spark is the trusted choice of millions of developers for their data engineering and machine learning needs. 

HDInsight on AKS offers these popular workloads with a common authentication model, shared meta store support, and prebuilt integrations which make it easy to deploy analytics applications.

Managed service reduces complexity

HDInsight on AKS is a managed service in the Azure Kubernetes Service infrastructure. With a managed service, customers aren’t burdened with the management of infrastructure and other software components, including operating systems, AKS infrastructure, and open-source software. This ensures that enterprises can benefit from ongoing security and functional and performance enhancements without investing precious development hours.  

Containerization enables seamless deployment, scaling, and management of key architectural components. The inherent resiliency of AKS allows pods to be automatically rescheduled on newly commissioned nodes in case of failures. This means jobs can run with minimal disruptions to Service Level Agreements (SLAs). 

Customers combining multiple workloads in their data lakehouse need to deal with a variety of user experiences, resulting in a steep learning curve. HDInsight on AKS provides a unified experience for managing their lakehouse. Provisioning, managing, and monitoring all workloads can be done in a single pane of glass. Additionally, with managed services for Prometheus and Grafana, administrators can monitor cluster health, resource utilization, and performance metrics.  

Through the autoscale capabilities included in HDInsight on AKS, resources—and thereby cost—can be optimized based on usage needs. For jobs with predictable load patterns, teams can schedule the autoscaling of resources based on a predefined timetable. Graceful decommission enables the definition of wait periods for jobs to be completed before ramping down resources, elegantly balancing costs with experience. Load-based autoscaling can ramp resources up and down based on usage patterns measured by compute and memory usage. 

HDInsight on AKS marks a shift away from traditional security mechanisms like Kerberos. It embraces OAuth 2.0 as the security framework, providing a modern and robust approach to safeguarding data and resources. In HDInsight on AKS authorization, access controls are based on managed identities. Customers can also bring their own virtual networks and associate them during cluster setup, increasing security and enabling compliance with their enterprise policies. The clusters are isolated with namespaces to protect data and resources within the tenant. HDInsight on AKS also allows management of cluster access using Azure Resource Manager (ARM) roles. 

Customers who’ve participated in the private preview love HDInsight on AKS. 

Here’s what one user had to say about his experience. 

“With HDInsight on AKS, we’ve seamlessly transitioned from the constraints of our in-house solution to a robust managed platform. This pivotal shift means our engineers are now free to channel their expertise towards core business innovation, rather than being entangled in platform management. The harmonious integration of HDInsight with other Azure products has elevated our efficiency. Enhanced security bolsters our data’s integrity and trustworthiness, while scalability ensures we can grow without hitches. In essence, HDInsight on AKS fortifies our data strategy, enabling more streamlined and effective business operations.”

Matheus Antunes, Data Architect, XP Inc

Source: microsoft.com

Continue reading

Azure high-performance computing powers energy industry innovation

The rising demand for energy

Global energy demand has rapidly increased over the last few years and looks set to continue accelerating at such a pace. With a booming middle class, economic growth, digitization, urbanization, and increased mobility of populations, energy suppliers are in a race to leverage the development of new technologies that can more optimally and sustainably generate, store, and transport energy to consumers.

With the impact of climate change adding urgency to minimizing energy waste, in addition to optimizing power production leaders in the renewable energy as well as oil and gas industries are accelerating sector-wide innovation initiatives that can drive differentiated impact and outcomes at scale.

As the population of developing countries continues to expand, the energy needs of billions of additional people in rural and especially urban areas will need to be catered to. McKinsey estimates that global energy consumption will triple by 2050, with oil and gas accounting for 65 percent of power consumption by then.

In addition, supplies of conventional oil and gas are also expected to decline in the not-too-distant future, shrinking in concentration to mostly the Middle East (oil) and countries like Russia, Iran, and Qatar (gas). As a result, the transition to more sustainable sources of power is leading global energy producers to leverage next-generation technologies to transform their solutions while simultaneously optimizing their operations.

New innovators in the renewable energy industry are also adopting next-generation technologies such as artificial intelligence (AI), advanced analytics, 3-D imaging, and the internet of things (IoT), supported by high-performance computing (HPC) capabilities, to maximize energy production and ensure a smoother transition to a more sustainable path.

Optimizing operational excellence in the energy industry

Instead of investing in complex, costly, and time-intensive on-premises resources, global energy leaders are leveraging the power of cloud capabilities such as Azure HPC + AI, to simulate highly complex, large-scale models and visualize seismic imaging and modeling, resulting in huge economic gains.

One of the key innovations enabling this strategic advantage is the dynamic scaling capability of Azure HPC + AI, powered by GPUs, which are ideal for running remote visualization, optimized virtual machines, and can be augmented with deep learning and predictive analytics, allowing customers to have on-demand intelligent computing to solve complex problems and drive tangible business outcomes.

Energy multinational bp, for example, believes technology innovation is the key to making a successful transition to net zero. The company chose to create digital twins to find opportunities for optimization and carbon reduction.

Drawing on over 250 billion data signals from an IoT network spanning bp's global operating assets, the company identified various opportunities to scale the digital twin solution to its entire operating base and reduce emissions by as much as 500,000 tons of CO2 equivalent every year.

Going green—Energy industry innovation abounds

The green energy sector is also grabbing hold of the opportunity presented by these exponential technologies to speed up the journey toward a more sustainable energy ecosystem.

Italian energy infrastructure operator Snam is harnessing Azure AI and a full stack of Azure IoT services to reduce carbon emissions and help meet its net-zero targets. Energy efficiency is top of the company's agenda. Snam aims to cut methane emissions by 55 percent by 2025, reach net zero by 2040, and exclusively transport decarbonized gas by 2050.

With any leakage in its operations posing a threat to field workers, maintenance staff, and people living near their network—not to mention the environment—Snam deployed an IoT network for real-time monitoring and to enhance its data collection and processing capabilities.

For wind energy solutions provider Vestas Wind Systems, a combination of Azure HPC and partner Minds.ai's machine learning platform, DeepSim, helped its wind farms mitigate the wake effect, generate more energy, and build a sustainable energy future.

Drawing on the Azure HBv3 virtual machines using third-generation AMD EPYCTM processors, Vestas can scale up and run millions of complex simulations that inform how controllers adjust turbines to optimize energy production.

The computing power offered by the AMD-based Azure HBv3 nodes allows Vestas to drive efficiencies that have the potential to unlock significantly more power and higher profits for wind farm operators by minimizing the estimated 10 percent of wind energy that is lost to wake effects.

Key takeaways

As the energy industry eyes a period of unprecedented growth and change, the role of technology will become ever more profound.

Leveraging powerful Microsoft Cloud capabilities such as HPC, AI, advanced analytics, big data, and IoT, the integrated advanced technology capabilities that have previously been the reserve of only a handful of the largest companies are now truly available to anyone.

Supported by these powerful next-generation technologies, energy companies can unlock greater efficiency, innovation, and growth to achieve gains across their operations and drive the world towards a brighter energy future.

Source: microsoft.com

Continue reading

RoQC and Microsoft simplify cloud migration with Microsoft Energy Data Services

The vast amount of data in energy companies slows down their digital transformation. Together with RoQC solutions, Microsoft Energy Data Services will accelerate your journey in democratizing access to data by providing an easy-to-deploy managed service fully supported by Microsoft.

Managing large data sets is complicated, and few industries have larger and more complex data sets than the energy industry. Data complexity and large investments in on-premises storage solutions and multitudes of computer systems prevent the transition to cloud-based sub-surface data management. A single company can have tens of petabytes of structured and unstructured data, which if not quality-assured, can lead to an increase in cost if it goes wrong.

Solutions from RoQC, a Norwegian software company, clean up structured data for energy companies. This makes data management more efficient from a time and cost perspective, and also makes decision-making more reliable.

With Microsoft Energy Data Services, energy companies can leverage new cloud-based data management capabilities provided by RoQC and Microsoft Energy Data Services.

Microsoft Energy Data Services is a data platform fully supported by Microsoft, that enables efficient data management, standardization, liberation, and consumption in energy exploration. The solution is a hyperscale data ecosystem that leverages the capabilities of the OSDU Data Platform™ and Microsoft's secure and trustworthy cloud services with our partners’ extensive domain expertise.

"Through machine learning, our software gives energy companies complete control of their data and assets. When the amounts of data are reduced, we eliminate uncertainty and duplication, and optimize the quality of the data sets. Traditionally a petrophysicist might spend a day or two cleaning up the logs for one well before they can be used for detailed analysis—with RoQC LogQA the same petrophysicist can clean hundreds of thousands of logs in the same timeframe. By cooperating with one of the largest platform providers in the world, we gain access to technology, competency, and markets it would be hard for us to get otherwise."—Bjørn Thorsen, CEO of RoQC.

New possibilities through cooperation

RoQC, a certified independent software vendor with Microsoft, has been able to expand its technology globally through the partnership.

Partner development manager for Microsoft Norway, Ole Christian Smerud, assures that the cooperation is mutually beneficial. "As a platform provider, we depend on strong partners to give our customers the best solutions. While we provide a platform, cloud competency, and access to an ecosystem for RoQC, they bring domain knowledge and relevance to their industry," he says.

Save millions with better data

RoQC believes that the energy industry struggles to take the step into the cloud, simply because of the data complexity and that most companies lack control over their data. By qualifying and quantifying data sets by identifying and deleting duplicates, RoQC Tools can reduce the data set size with commensurate dramatic savings in storage costs.

By reducing the amount of data by 10 to 30 percent, we’re talking millions of dollars in savings. The bigger the organization, the bigger the effect.

RoQC Tools are primarily designed so that data managers can perform tasks that are usually time-consuming as efficiently as possible. Very often they can complete a task that usually takes months, in a minute or two. Sometimes, the tasks would not be possible at all without the tools.

There is an obvious and well-documented correlation between increasing the quality of your data and reducing the risk of decisions based on that data. Geoscientists and project leaders in this field make decisions worth millions, maybe billions. You don’t want to make a decision of that magnitude based on insufficient or weak data.

RoQC believes the energy companies’ data is the key to shifting away from fossil resources. In the data sets, subsea energy companies have knowledge of "everything" about the ocean floor and sub-sea.

"Minerals from the ocean floor and sub-surface might be the next big thing for subsea oil-dependent nations like Norway. It is an already overused statement, but data is literally the new oil for this industry," says Bjørn Thorsen.

Preparing efficient data migration

RoQC provides both tools and consultants to enable a client to prepare their data prior to migrating the data to Azure. This preparation can include everything from simply identifying and removing duplicates to developing and implementing standards and then cleaning the data to comply with the standards. These preparations can be done directly in the clients’ normal (e.g., Halliburton/Schlumberger) interpretation platforms.

Furthermore, RoQC’s LogQA provides extremely powerful native, machine learning–based QA and cleanup tools for log data once the data has been migrated to Microsoft Energy Data Services, an enterprise-grade OSDU Data Platform on the Microsoft Cloud.

LogQA monitors the quality of the well log data that a client has stored on OSDU Data Platform. LogQA was partially developed in collaboration with Microsoft as part of Microsoft Energy Data Services, and LogQA is maintained on the latest OSDU Data Platform APIs and version/schema.

As LogQA is native to the Microsoft Cloud infrastructure there is no customer deployment required before a customer can utilize LogQA to monitor, identify, and rapidly rectify the data quality issues. LogQA is designed to work with typically energy industry client datasets, which is potentially millions of well logs.

How to work with RoQC Solutions on Microsoft Energy Data Services

For access to RoQC solutions, reach out to Bjørn Thorsen, CEO, RoQC Data Management AS, Norway at Bjorn@roqc.no.

Source: microsoft.com

Continue reading

EPAM and Microsoft partner on data governance solutions with Microsoft Energy Data Services

The energy industry creates and consumes large amounts of highly complex data for key business decisions, like where to drill the next series of wells, how to optimize production, and where to lease the next big field. Despite good intentions, the industry is still plagued by large quantities of data that are inconsistent in location, quality, and format—much of which cannot reliably be found or used when needed. Even when the data is reliable, it can be locked into application-specific data stores that limit its use. The solution to this dilemma is multi-faceted and increasingly includes cloud technology, the OSDU™ Data Platform, modern applications, and data governance focused on people and their business processes.

Microsoft Energy Data Services is a data platform fully supported by Microsoft, that enables efficient data management, standardization, liberation, and consumption in energy exploration. The solution is a hyperscale data ecosystem that leverages the capabilities of the OSDU Data Platform, Microsoft's secure and trustworthy cloud services with our partners’ extensive domain expertise.

Cloud and the OSDU Data Platform

Cloud-based computing is the future—scalable, reliable, secure storage and compute capabilities, all managed for you with many powerful add-on capabilities at your fingertips. For the energy industry, the Open Group® OSDU Data Platform is rapidly emerging as the standard—an open source, cloud-based data platform that unlocks data from applications and provides standard data schemas and access protocols, enabling both data governance and rapid innovation.

One of the things that EPAM discovered when delivering app developer boot camps and deploying the platform for ourselves and for clients is its high level of complexity. In those earlier days, platform deployment was a multi-step process, with each service being deployed and validated separately, taking up to a week. Before we could move on to solving business problems, a part of our work was to guide our clients through various technical deployment obstacles. In addition, it took another several days to ingest pre-formatted sample data in order to test the platform with real data. Not anymore.

Microsoft Energy Data Services

Microsoft has made the OSDU Data Platform enterprise-ready and pre-bundled with the capabilities needed to optimize Energy Company data value using the Microsoft Cloud. EPAM has seen its benefits. As an enterprise-grade platform, Microsoft Energy Data Services has nearly single-click deployment. Deployment time has reduced significantly—what previously took multiple days now takes about 45 minutes! Similarly, the time to ingest the sample data is drastically reduced from one week to around one hour! In addition, the management layer surrounding the platform provides the assured reliability, stability, security, tools, performance, and the SLAs needed by large enterprises such as major energy companies.

Data governance and modern applications

As noted before, excellent infrastructure alone does not magically solve all data and business problems. With Microsoft Energy Data Services providing a solid foundation with which to store data, process data, and build and host cloud-native apps aligned with the OSDU Technical Standard, what remains to empower a data-driven organization is modern applications and data governance.

Read More: PL-400: Microsoft Power Platform Developer

It is a daunting task to manually track the manifold ways that data enters the company, the many places it is stored, and the many ways it is consumed, enriched, and duplicated. Improving this requires a team who can map out the detailed way in which all of this happens today. It also takes modern digital tools to automate the aggregation, parsing, quality assessment, and lineage-tracking of the data. It takes people with a broad and deep view to accomplish this for large organizations—people who understand the business, the data types, the technology, and how to provide the right data, in the right formats, in the right place, at the right time, to the right people. That includes application connectors and analytical applications themselves designed for the modern cloud environment so that liberated data can move back and forth to users seamlessly.

How to work with EPAM on Microsoft Energy Data Services

EPAM brings industry knowledge, technical expertise, tools, frameworks, relationships with software vendors, and world-class delivery built on the Microsoft Energy Data Services platform. EPAM has developed a document extraction and processing system (DEPS) accelerator, which provides capabilities to facilitate the development of customizable workflows for extracting and processing unstructured data in the form of scanned or digitalized document formats. DEPS is powered by Azure AI/machine learning and deep learning algorithms.  It includes pluggable sub-systems for customization, uses machine learning pre/post processors, validation and extensions for UI review, automation machine learning models training, manual labeling, and analytics capabilities to improve classic optical character recognition (OCR) and text extraction accuracy. DEPS can be adapted to process numerous data types covering both image and text, PDF, XLS, ASCII, and other file formats.

Microsoft Energy Data Services is an enterprise-grade, fully-managed, OSDU Data Platform for the energy industry that is efficient, standardized, easy to deploy, and scalable for data management—for ingesting, aggregating, storing, searching, and retrieving data. The platform will provide scale, security, privacy, and compliance expected by our enterprise customers. EPAM offers services providing the right data, in the right formats, in the right place, at the right time, to the right people, which includes application connectors and analytical applications, with data contained in Microsoft Energy Data Services.

Source: microsoft.com

Continue reading

Elevate your visualizations with Azure Managed Grafana—now generally available

As part of our continued commitment to open source solutions, we are announcing the general availability of Azure Managed Grafana, a managed service that enables you to run Grafana natively within the Azure cloud platform. With Azure Managed Grafana, you can seamlessly and securely connect with and scale to businesses’ existing Azure services, enhancing observability and cloud management.

In addition to the features announced during preview, with general availability, we’re introducing new capabilities that include the latest Grafana v9.0 features with its improved alerting experience as well as zone redundancy (in preview) and API key support.

New connections and integrations with Azure services

With general availability, we are adding new integrations with Azure services, allowing you to realize the benefits of Grafana as efficiently and effectively as possible.

We have introduced several new out-of-the-box dashboards for Azure Monitor. For example, with Availability Tests Geo Map dashboard for Azure Monitor application insights, you can view the results and responsiveness of your application availability tests based on geographic location. Additionally, with the new out-of-the-box Load Balancing dashboard for Azure Monitor network insights, you can monitor key performance metrics for all your Azure load balancing resources, including Load Balancers, Application Gateway, Front Door, and Traffic Manager.

The new “pin to Grafana” feature for Azure Monitor Logs allows you to seamlessly add charts and queries from Azure Monitor Logs to Grafana dashboards with just one click. In the illustration below, you can see how the Azure Monitor Logs query on the left is replicated in the Grafana interface on the right.

Similarly, we have introduced new out-of-the-box dashboards for Azure Container Apps as well. The new Aggregate View dashboard for Azure Container Apps depicts a geographic map of your container apps filtered by resource group, environment, and region with drill-down links to a detailed dashboard for each app. The new App View dashboard for Azure Container Apps monitors the performance of Azure Container Apps by viewing the key metrics of CPU, memory, restarts, and network traffic or by revision, replica, and status code.

Source: microsoft.com

Continue reading

Azure Data Explorer: Log and telemetry analytics benchmark

Azure Data Explorer (ADX), a component of Azure Synapse Analytics, is a highly scalable analytics service optimized for structured, semi-structured, and unstructured data. It provides users with an interactive query experience that unlocks insights from the ocean of ever-growing log and telemetry data. It is the perfect service to analyze high volumes of fresh and historical data in the cloud by using SQL or the Kusto Query Language (KQL), a powerful and user-friendly query language.

Azure Data Explorer is a key enabler for Microsoft’s own digital transformation. Virtually all Microsoft products and services use ADX in one way or another; this includes troubleshooting, diagnosis, monitoring, machine learning, and as a data platform for Azure services such as Azure Monitor, PlayFab, Sentinel, Microsoft 365 Defender, and many others. Microsoft’s customers and partners are using ADX for a large variety of scenarios from fleet management, manufacturing, security analytics solutions, package tracking and logistics, IoT device monitoring, financial transaction monitoring, and many other scenarios. Over the last years, the service has seen phenomenal growth and is now running on millions of Azure virtual machine cores.

Last year, the third generation of the Kusto engine (EngineV3) was released and is currently offered as a transparent, in-place upgrade to all users not already using the latest version. The new engine features a completely new implementation of the storage, cache, and query execution layers. As a result, performance has doubled or more in many mission-critical workloads.

Superior performance and cost-efficiency with Azure Data Explorer

To better help our users assess the performance of the new engine and cost advantages of ADX, we looked for an existing telemetry and logs benchmark that has the workload characteristics common to what we see with our users:

1. Telemetry tables that contain structured, semi-structured, and unstructured data types.

2. Records in the hundreds of billions to test massive scale.

3. Queries that represent common diagnostic and monitoring scenarios.

As we did not find an existing benchmark to meet these needs, we collaborated with and sponsored GigaOm to create and run one. The new logs and telemetry benchmark is publicly available in this GitHub repo. This repository includes a data generator to generate datasets of 1GB, 1TB, and 100TB, as well as a set of 19 queries and a test driver to execute the benchmark.

The results, now available in the GigaOm report, show that Azure Data Explorer provides superior performance at a significantly lower cost in both single and high-concurrency scenarios. For example, the following chart taken from the report displays the results of executing the benchmark while simulating 50 concurrent users: 

Source: microsoft.com

Continue reading

Azure Orbital Ground Station as Service extends life and reduces costs for satellite operators

How can Microsoft empower satellite operators to focus on their mission and enable them to continue the operation of their satellites, without making capital investments in their ground infrastructure?

To answer that question, Microsoft worked alongside the National Oceanic and Atmospheric Administration (NOAA), and our partner Xplore, to demonstrate how the commercial cloud can provide satellite mission management for NOAA’s legacy polar satellites (NOAA-18)—extending the mission life of these satellites while reducing the cost of operation through Azure Orbital Ground Station as-a-Service (GSaaS).

Partnering with the National Oceanic and Atmospheric Administration and Xplore

The initiative was part of a year-long cooperative research and development agreement (CRADA) with NOAA, where we worked together to determine the ability of the Azure Orbital platform to connect and downlink data from NOAA satellites. NOAA also tested the ability of Microsoft Azure to comply with specified security controls in a rapid and effective manner. Our cloud-based solutions performed successfully across all measures.

Partners are central to Microsoft’s approach to space, and they played a key role in this project. As part of the CRADA, we leveraged our partner network to bring together Azure Orbital with Xplore’s Major Tom mission control software platform. This approach enabled NOAA to transmit commands to the NOAA-18 spacecraft and verify the receipt of these commands. This test was conducted in real-time, and data was flowing bi-directionally with the NOAA-18 satellite.

Commercial technology enabled the rapid demonstration of these innovative capabilities. Xplore was able to move quickly to bring functions of NOAA’s heritage space system architecture to the Azure cloud through their Major Tom platform. This highlights the power of Azure as a platform to bring together Azure Orbital as the ground station, Major Tom to provide the mission control software for commanding and telemetry viewing, and the NOAA operators to monitor the scenarios.

This successful demonstration shows that the Azure Orbital GSaaS, and the partner network it brings together, enables sustainable outcomes for satellite operators. Our work with NOAA is just the beginning of the journey. We look forward to partnering with additional satellite operators to help them reduce their infrastructure management costs, lower latency, increase capacity and resiliency, and empower their missions through the power of Azure Orbital GSaaS and the Azure cloud.

Learn more about Azure Orbital and Azure Space

To learn more about Azure Orbital GSaaS, visit our product page, or take a look at the session with Microsoft Mechanics, which goes into more detail on how we connect space satellites around the world and bring earth observational data into Azure for analytics via Microsoft and partner ground stations. We demonstrate how it works and how it fits into Microsoft’s strategy with Azure Space to bring cloud connectivity everywhere on earth and to make space satellite data accessible for everyday use cases.

More broadly, Azure Space marks the convergence between global satellite constellations and the cloud. As the two join together, our purpose is to bring cloud connectivity to even the most remote corners of the earth, connect to satellites, and harness the vast amount of data collected from space. This can help solve both long-term trending issues affecting the earth like climate change, or short-term real-time issues such as connected agriculture, monitoring and controlling wildfires, or identifying supply chain bottlenecks.

Source: microsoft.com

Continue reading

Enhance your data visualizations with Azure Managed Grafana—now in preview

Organizations are transforming their digital environments to increase agility and to operate more efficiently. We see this transformation in how customers migrate to the cloud and adopt cloud-native technologies and practices in their own environments. As their digital estates become increasingly more complex and critical to their business operations, it becomes even more important to effectively manage and monitor their applications and infrastructure.   

Grafana is a popular open-source analytics visualization tool that allows users to bring together logs, traces, metrics, and other disparate data from across an organization, regardless of where they are stored. Last year, we announced our strategic partnership with Grafana Labs to develop a Microsoft Azure managed service that lets customers run Grafana natively within the Azure cloud platform. Today, we are announcing that Azure Managed Grafana is available in preview. With Azure Managed Grafana, the Grafana dashboards our customers are familiar with are now integrated seamlessly with the services and security of Azure.

Seamless connection across Azure data sources and beyond

The Grafana application lets users easily visualize all their telemetry data in a single user interface. With Grafana's extensible architecture, users can visualize and correlate multiple data sources across on-premises, Azure, and multi-cloud environments. Azure Managed Grafana particularly optimizes this experience for Azure-native data stores such as Azure Monitor and Data Explorer thus making it easy for customers to connect to any resource in their subscription and view all resulting telemetry in a familiar Grafana dashboard.

Customers can preserve existing charts in the Azure portal that are used for monitoring. Through service-to-service integration, our customers can bring any chart in the Azure portal over to their Azure Managed Grafana instance with a one-click “pin to” operation thus automating the entire migration process. 

Azure Managed Grafana also provides a rich set of built-in dashboards for various Azure Monitor features to help customers easily build new visualizations. For example, some features with built-in dashboards include Azure Monitor application insights, Azure Monitor container insights, Azure Monitor virtual machines insights, and Azure Monitor alerts.

Secured access and sharing of Grafana dashboards with Azure Active Directory

In Azure Managed Grafana, customers can customize user permissions with specific roles and assignments stored in Azure Active Directory. These definitions are mapped transparently to Grafana’s internal roles, which enforces the actual access control. This integration enables both simplicity and consistency by allowing customers to manage users in their teams and authorize their use of a Grafana instance centrally through Azure Active Directory.

On the backend, Azure Managed Grafana can be configured to access Azure Monitor through a managed identity that was set up as part of the Grafana instance creation. Using this option, customers do not need to deal with another credential separately—though that is still possible if preferred.

Source: microsoft.com

Continue reading

Feathr: LinkedIn’s feature store is now available on Azure

Feature store motivation

With the advance of AI and machine learning, companies start to use complex machine learning pipelines in various applications, such as recommendation systems, fraud detection, and more. These complex systems usually require hundreds to thousands of features to support time-sensitive business applications, and the feature pipelines are maintained by different team members across various business groups.

In these machine learning systems, we see many problems that consume lots of energy of machine learning engineers and data scientists, in particular duplicated feature engineering, online-offline skew, and feature serving with low latency.

Figure 1: Illustration on problems that feature store solves.

Duplicated feature engineering

◉ In an organization, thousands of features are buried in different scripts and in different formats; they are not captured, organized, or preserved, and thus cannot be reused and leveraged by teams other than those who generated them.

◉ Because feature engineering is so important for machine learning models and features cannot be shared, data scientists must duplicate their feature engineering efforts across teams.

Online-offline skew

◉ For features, offline training and online inference usually require different data serving pipelines—ensuring consistent features across different environments is expensive.

◉ Teams are deterred from using real-time data for inference due to the difficulty of serving the right data.

◉ Providing a convenient way to ensure data point-in-time correctness is key to avoid label leakage.

Serving features with low latency

◉ For real-time applications, getting feature lookups from database for real-time inference without compromising response latency and with high throughput can be challenging.

◉ Easily accessing features with very low latency is key in many machine learning scenarios, and optimizations needs to be done to combine different REST API calls to features.

To solve those problems, a concept called feature store was developed, so that:

◉ Features are centralized in an organization and can be reused

◉ Features can be served in a synchronous way between offline and online environment

◉ Features can be served in real-time with low latency

Introducing Feathr, a battle-tested feature store

Developing a feature store from scratch takes time, and it takes much more time to make it stable, scalable, and user-friendly. Feathr is the feature store that has been used in production and battle-tested in LinkedIn for over 6 years, serving all the LinkedIn machine learning feature platform with thousands of features in production.

At Microsoft, the LinkedIn team and the Azure team have worked very closely to open source Feathr, make it extensible, and build native integration with Azure. It’s available in this GitHub repository and you can read more about Feathr on the LinkedIn Engineering Blog.

Some of the highlights for Feathr include:

◉ Scalable with built-in optimizations. For example, based on some internal use case, Feathr can process billions of rows and PB scale data with built-in optimizations such as bloom filters and salted joins.

◉ Rich support for point-in-time joins and aggregations: Feathr has high performant built-in operators designed for Feature Store, including time-based aggregation, sliding window joins, look-up features, all with point-in-time correctness.

◉ Highly customizable user-defined functions (UDFs) with native PySpark and Spark SQL support to lower the learning curve for data scientists.

◉ Pythonic APIs to access everything with low learning curve; Integrated with model building so data scientists can be productive from day one.

◉ Rich type system including support for embeddings for advanced machine learning/deep learning scenarios. One of the common use cases is to build embeddings for customer profiles, and those embeddings can be reused across an organization in all the machine learning applications.

◉ Native cloud integration with simplified and scalable architecture, which is illustrated in the next section.

◉ Feature sharing and reuse made easy: Feathr has built-in feature registry so that features can be easily shared across different teams and boost team productivity.

Feathr on Azure architecture

The high-level architecture diagram below articulates how would a user interacts with Feathr on Azure:

Figure 2: Feathr on Azure architecture.

1. A data or machine learning engineer creates features using their preferred tools (like pandas, Azure Machine Learning, Azure Databricks, and more). These features are ingested into offline stores, which can be either:

◉ Azure SQL Database (including serverless), Azure Synapse Dedicated SQL Pool (formerly SQL DW).

◉ Object storage, such as Azure BLOB storage, Azure Data Lake Store, and more. The format can be Parquet, Avro, or Delta Lake.

2. The data or machine learning engineer can persist the feature definitions into a central registry, which is built with Azure Purview.

3. The data or machine learning engineer can join on all the feature dataset in a point-in-time correct way, with Feathr Python SDK and with Spark engines such as Azure Synapse or Databricks.

4. The data or machine learning engineer can materialize features into an online store such as Azure Cache for Redis with Active-Active, enabling multi-primary, multi-write architecture that ensures eventual consistency between clusters.

5. Data scientists or machine learning engineers consume offline features with their favorite machine learning libraries, for example scikit-learn, PyTorch, or TensorFlow to train a model in their favorite machine learning platform such as Azure Machine Learning, then deploy the models in their favorite environment with services such as Azure Machine Learning endpoint.

6. The backend system makes a request to the deployed model, which makes a request to the Azure Cache for Redis to get the online features with Feathr Python SDK.

A sample notebook containing all the above flow is located in the Feathr repository for more reference.

Feathr has native integration with Azure and other cloud services. The table below shows these integrations:

Feathr component	Cloud Integrations
Offline store – Object Store	Azure Blob Storage Azure ADLS Gen2 AWS S3
Offline store – SQL	Azure SQL DB Azure Synapse Dedicated SQL Pools (formerly SQL DW) Azure SQL in VM Snowflake
Online store	Azure Cache for Redis
Feature Registry	Azure Purview
Compute Engine	Azure Synapse Spark Pools Databricks
Machine Learning Platform	Azure Machine Learning Jupyter Notebook
File Format	Parquet ORC Avro Delta Lake

Table 1: Feathr on Azure Integration with Azure Services.

Installation and getting started

Feathr has a pythonic interface to access all Feathr components, including feature definition and cloud interactions, and is open sourced here. The Feathr python client can be easily installed with pip:

pip install -U feathr

Source: microsoft.com

Continue reading

Key foundations for protecting your data with Azure confidential computing

The exponential growth of datasets has resulted in growing scrutiny of how data is exposed—both from a consumer data privacy and compliance perspective. In this context, confidential computing becomes an important tool to help organizations meet their privacy and security needs surrounding business and consumer data.

Confidential computing technology encrypts data in memory and only processes it once the cloud environment is verified, preventing data access from cloud operators, malicious admins, and privileged software such as the hypervisor. It helps keep data protected throughout its lifecycle—in addition to existing solutions of protecting data at rest and in transit, data is now protected while in use.

Thanks to confidential computing, organizations across the world can now unlock opportunities that were not possible before. For example, they can now benefit from multi-party data analytics and machine learning that combine datasets from parties that would have been unwilling or unable to share them, keeping data private across participants. In fact, RBC created a platform for privacy-preserving analytics for customers to opt-in for more optimized discounts. The platform generates insights into consumer purchasing preferences by confidentially combining RBC’s credit and debit card transactions with retailer data of what specific items consumers purchased.

Industry leadership and standardization

Microsoft has long been a thought leader in the field of confidential computing. Azure introduced “confidential computing” in the cloud when we became the first cloud provider to offer confidential computing virtual machines and confidential containers support in Kubernetes for customers to run their most sensitive workloads inside Trusted Execution Environments (TEEs). Microsoft is also a founding member of the Confidential Computing Consortium (CCC), a group that brings together hardware manufacturers, cloud providers, and solution vendors to jointly work on ways to improve and standardize data protection across the tech industry.

Confidential computing foundations

Our bar for confidentiality aligns and extends the bar set by the CCC to provide a comprehensive foundation for confidential computing. We strive to provide customers the technical controls to isolate data from Microsoft operators, their own operators, or both. In Azure, we have confidential computing offerings that go beyond hypervisor isolation between customer tenants to help protect customer data from access by Microsoft operators. We also have confidential computing with secure enclaves to additionally help prevent access from customer operators.

Our foundation for confidential computing includes:

◉ Hardware root-of-trust to ensure data is protected and anchored into the silicon. Trust is rooted to the hardware manufacturer, so even Microsoft operators cannot modify the hardware configurations.

◉ Remote attestation for customers to directly verify the integrity of the environment. Customers can verify that both the hardware and software on which their workloads run are approved versions and secured before allowing them to access data.

◉ Trusted launch is the mechanism that ensures virtual machines boot with authorized software and that uses remote attestation so that customers can verify. It’s available for all VMs including confidential VMs, bringing Secure Boot and vTPMs, to add defense against rootkits, bootkits, and malicious firmware.

◉ Memory isolation and encryption to ensure data is protected while processing. Azure offers memory isolation by VM, container, or application to meet the various needs of customers, and hardware-based encryption to prevent unauthorized viewing of data, even with physical access in the datacenter.

◉ Secure key management to ensure that keys stay encrypted during their lifecycle and release only to the authorized code.

The above components together form the foundations for what we consider to be confidential computing. And today, Azure has more confidential compute options spanning hardware and software than any other cloud vendor.

Innovative new hardware

Our new Intel-based DCsv3 confidential VMs include Intel SGX that implement hardware-protected application enclaves. Developers can use SGX enclaves to reduce the amount of code that has access to sensitive data to a minimum. Additionally, we will enable Total Memory Encryption-Multi-Key (TME-MK) so that each VM can be secured with a unique hardware key.

Our new AMD-based DCasv5/ECasv5 confidential VMs available provide Secure Encrypted Virtualization-Secure Nested Paging (SEV-SNP) to provide hardware-isolated virtual machines that protect data from other virtual machines, the hypervisor, and host management code. Customers can lift and shift existing virtual machines without changing code and optionally leverage enhanced disk encryption with keys they manage or that Microsoft manages.

To support containerized workloads, we are making all of our confidential VMs available in Azure Kubernetes Service (AKS) as a worker node option. Customers can now protect their containers with Intel SGX or AMD SEV-SNP technologies.

Azure’s memory encryption and isolation options provide stronger and more comprehensive protections for customer data than any other cloud.

Customer successes across industries

Many organizations are already leveraging the great data privacy and security benefits of Azure confidential computing.

Secure AI Labs have been using a private preview of our AMD-based virtual machines to create a platform where healthcare researchers can more easily collaborate with healthcare providers to advance research. Luis Huapaya, VP of Engineering at Secure AI Labs Inc, mentions, “Because of Azure confidential computing, Secure AI Labs can realize all of the benefits of running in Azure without ever sacrificing on security. One could argue that running a virtual payload within Azure confidential computing might be more secure than running in a private server on-premise. It also offers remote attestation, a pivotal security feature because it provides a virtual payload the ability to give cryptographic proof of its identity and verify its running inside an enclave. Azure confidential computing with AMD SEV-SNP makes our job a lot easier.”

While regulated industries have been the early adopters due to compliance needs and highly sensitive data, we are seeing growing interest across industries, from manufacturing to retail and energy, for example.

Signal Messenger, a worldwide messaging app known for high security and privacy, leverages Azure confidential computing with Intel SGX to protect customer data, such as contact info. Jim O’Leary, VP of Engineering at Signal says, “To meet the security and privacy expectations of millions of people every day, we utilize Azure confidential computing to provide scalable, secure environments for our services. Signal puts users first, and Azure helps us stay at the forefront of data protection with confidential computing.”

We are excited to see organizations bring more workloads to Azure with confidence in the data protection of Azure confidential computing to meet the privacy needs of their customers.

Azure confidential cloud

Azure is the world’s computer from cloud to edge. Customers of all sizes, across all industries, want to innovate, build, and securely operate their applications across multi-cloud, on-premises, and edge. Just as HTTPS has become pervasive for protecting data during internet web browsing, here at Azure, we believe that confidential computing will be a necessary ingredient for all computing infrastructure. 

Our vision is to transform the Azure cloud into the Azure confidential cloud, moving from computing in the clear to computing confidentially across the cloud and edge. We want to empower customers to achieve the highest levels of privacy and security for all their workloads.

Alongside our $20 billion investment over the next five years in advancing our security solutions, we will partner with hardware vendors and innovate within Microsoft to bring the highest levels of data security and privacy to our customers. In our journey to become the world’s leading confidential cloud, we will drive confidential computing innovations horizontally across our Azure infrastructure and vertically through all the Microsoft services that run on Azure.

Source: microsoft.com

Continue reading

Transform your business with Microsoft's unrivaled end-to-end data platform

Businesses today are experiencing the most significant change in our economy in a generation and the ability to realize the full potential of the cloud through digital transformation has never been greater. There are three core capabilities businesses need that together drive successful transformation: limitless database scale and performance, unmatched analytics and insights, and unified data governance.

Today we shared a number of announcements to help organizations evolve rapidly even in the face of sustained uncertainty. Across industries, companies are investing in these core capabilities to ensure they have the flexibility to innovate anywhere and meet evolving business needs.

Limitless database scale and performance

We are announcing the preview of SQL Server 2022, the most Azure-enabled SQL Server release yet. This enables easier cloud integration than ever before with new disaster recovery functionality for Azure SQL Managed Instance and seamless analytics over on-premises operational data with Synapse Link for SQL Server. Microsoft continues to invest in our Azure SQL family of databases providing flexible options for app migration, modernization, and development. Azure SQL Managed Instance has a number of key announcements, including business offers with significantly more memory per vCore and increased storage to 16 TB in both the general purpose and business-critical service tiers, providing more room for app growth.

Azure is also the best cloud for NoSQL and open source databases. Azure Cosmos DB is a fully managed NoSQL database built to support production applications at any size or scale. Financially backed service-legal agreements (SLA's) guaranteeing consistency, high throughput, less than 10 ms latency reads and writes, and 99.999 percent availability make it simple for developers to independently and elastically scale applications across any Azure region. Today we are announcing the general availability of Azure Managed Instance for Apache Cassandra, providing the ability to provision managed native Apache Cassandra clusters with automated deployment and scaling operations, accelerating hybrid scenarios, and reducing ongoing maintenance. Flexible Server (preview), a new deployment option for Azure Database for MySQL is also now generally available. With Flexible Server, customers can benefit from a fully managed service designed to provide maximum control of your databases, high availability options to ensure zero data loss, built-in capabilities for cost optimization, and increased productivity enabled by the Azure ecosystem.

Unmatched analytics and insights

When it comes to analytics and insights, Azure is simply unmatched. When we debuted Azure Synapse Analytics, we effectively removed the barriers between enterprise data warehousing and big data analytics to enable data professionals to collaborate, build, and manage their analytic solutions with ease. Since its launch, the number of Azure customers running petabyte-scale workloads has increased fivefold. 

But we didn’t stop there. We introduced Azure Synapse Link for Azure Cosmos DB last year, creating a simple, low-cost, cloud-native HTAP implementation that enables immediate, in-the-moment business insights. We continue our investments today with the general availability of Azure Synapse Link for Dataverse, enabling immediate insights with high-value Dynamics 365 data, and a preview of Azure Synapse Link for SQL Server 2022. We are also announcing that Azure Synapse Data Explorer is joining the existing SQL and Apache Spark analytical runtimes in Azure Synapse—a highly scalable engine to automatically index structured, semi-structured, and free-text data commonly found in telemetry, time series, and logs to surface enriched insights that can directly improve business decisions. 

Unified data governance

The need for a comprehensive data governance service has never been stronger. Azure Purview, our unified data governance service, helps businesses manage and govern their on-premises, multi-cloud, and software as a service (SaaS) data. It’s exciting to see the strong interest in Azure Purview since its general availability, with over 57 billion data assets discovered by customers already. Azure Purview ensures your Microsoft data estate is governed through deep integrations with data services in Azure, Microsoft 365, Microsoft Power Platform, and more. Every organization can now build a unified data governance solution to maximize the value of their data in the cloud.

Customers like the London Heathrow Airport, for example, have already achieved tremendous success with Azure Purview. The airport’s data teams use lineage data in Azure Purview to find correlations between datasets which is critical in determining how each airport service affects broader operations.

“We used to have a lot of data about things we generally already understood but now that we’ve adopted Azure Purview, we have insight into the unknown, meaning we can aggregate multiple data sources in a more user-friendly way to discover where we can create efficiencies and make better predictions.”—Dave Draffin, Azure Cloud and Data Architect, Heathrow Airport

Source: microsoft.com

Continue reading

Join Microsoft at ISC 2021 and drive innovations with Azure high-performance computing

Customers around the world rely on Microsoft Azure to drive innovations.

Whether it’s understanding our environment, public health, energy sustainability, weather modeling, economic growth, and many others finding solutions to these important challenges requires huge amounts of focused computing power. Customers are increasingly finding the best way to access such high-performance computing (HPC) is through the agility, scale, security, and leading-edge performance of Azure’s purpose-built HPC and AI cloud services.

Read More: MB-330: Microsoft Dynamics 365 Supply Chain Management

Azure’s market-leading vision for HPC and AI is based on a core of genuine and recognized HPC expertise, using proven HPC technology and design principles, enhanced with the best features of the cloud. The result is a complete platform that delivers performance, scalability, and value, unlike any other cloud. This means applications scaling 12 times higher than other public clouds. It means higher application performance per VM. It means powering AI workloads for one customer with a supercomputer fit to be among the top five in the world. And it means delivering massive computing power into the hands of medical researchers over a weekend to prove out life-saving innovations in the fight against COVID-19.

Join us at ISC 2021 to learn more about Azure HPC and AI Supercomputing and Quantum

Microsoft will deliver a keynote at the International Supercomputing Conference (ISC) 2021 on quantum computing, and participate in multiple places across the ISC 2021 technical program, including panel sessions, talks, and presentations by our customers.

Additionally, we invite you to join us at an informal off-hour event at ISC on Monday, June 28, 2021, from 6:30 PM – 7:30 PM CEST for our inaugural “open community social.” We will have a range of HPC and Quantum experts from research, engineering, and customer-facing roles. We will also officially launch our Azure HPC community program. Learn additional details about the Azure HPC after-hour social event at ISC 2021 and join our Discord group prior to the off-hour event.

Announcements for Azure HPC and AI Supercomputing in 2021

Azure FX-series Virtual Machine general availability announcement

We are announcing the general availability of Azure FX-series Virtual Machine, specifically designed to optimize electronic design automation (EDA) workload processing with a purpose-built virtual machine architected for EDA workloads.

NVIDIA ND A100 v4 launch

Azure announced the general availability of scale-out NVIDIA A100 GPU Clusters with HDR 200 Gb/s InfiniBand networking: the fastest public cloud supercomputer. The ND A100 v4 VM is powered by NVIDIA A100 Tensor Core GPUs and is designed to let our most demanding customers scale up and scale out without ever slowing down.

Xilinx NP launch

Azure launched general availability of the Azure NP-Series Virtual Machine. The NP-series virtual machines are powered by Xilinx U250 FPGAs for accelerating workloads including machine learning inference, video transcoding, and database search and analytics. NP-series VMs are also powered by Intel Xeon 8171M (Skylake) CPUs with an all-core turbo clock speed of 3.2 GHz.

AMD HBv3 launch

Azure announced the general availability of the Azure HBv3 virtual machine for HPC workloads advancing the velocity at which we bring the latest technologies to our HPC customers and the compute performance we put at their fingertips.

Azure HPC and AI Collaboration Centers launch

Microsoft announced the HPC and AI Collaboration Centers program with five inaugural partners. These partners will share best practices for unlocking customer innovation and productivity with HPC and AI.

Big moments for Azure HPC and AI Supercomputing in 2021

UK Met Office supercomputing announcement

The UK Met Office supercomputing announcement highlighted that Microsoft had been awarded a contract to deliver a ten-year managed supercomputing service, providing advanced supercomputing capabilities for weather and climate research, ensuring the continuation of the Met Office’s leadership in this area.

Jellyfish

Jellyfish Pictures needed to enable secure remote access to immense computing power to render visual effects and animation sequences. The studio gained burst rendering on up to 90,000 processor cores in the cloud with Microsoft Azure Virtual Machine Scale Sets, HBv2 virtual machines, and Avere vFXT for Azure. It takes 80 percent off those rendering costs with Azure Spot Virtual Machines and uses an Azure ExpressRoute connection to minimize latency while more securely managing storage in one place, without replication.

Van Gogh Museum

In an average year, tens of thousands of Chinese visitors head to the Netherlands to tour the Van Gogh Museum. In recognition of this interest, the museum has created a mini-program for the social media platform WeChat. Through this app-like program, Chinese users can take selfies and have them digitally reworked in the style of Vincent Van Gogh. To remain compliant with GDPR legislation, and to scale the mini-program to meet demand, the museum has deployed it in Microsoft Azure with Azure Blob Storage, Azure Functions, and Azure Kubernetes Service (AKS).

Wildlife Protection Agency

Wildlife Protection Solutions (WPS) sits in the nexus between the most remote places in the world and burgeoning animal conservation science. It provides the monitoring technology that conservation groups need to keep watch over wild places and protect wildlife. Conservationists use remote cameras to gather image data about the status of the species they protect, but the number of images that must be analyzed before action can be taken is overwhelming. WPS overcomes this barrier in collaboration with Microsoft AI for Earth, supported by Azure technologies for a species-preserving solution.

Source: microsoft.com

Continue reading