Reflecting on 2023—Azure Storage

The beginning of a new year often prompts reflection along with planning for the future. At the forefront of our priorities lies the commitment to enhance the Azure platform and its ecosystem, fuel groundbreaking AI solutions, and facilitate seamless migrations to the cloud. We achieve this through purpose-built storage solutions tailored to meet the unique demands of your workloads.

2023 was an emblematic year of growth for storage and data services with AI at the top of mind for many customers. Azure customers’ overall data estate continued to grow, powered by the emergence of new workloads and access patterns. Notable Azure Storage figures include:

• The storage platform now processes more than 1 quadrillion (that’s 1000 trillion!) transactions a month with over 100 exabytes of data read and written each month. Both numbers were sharply higher compared to the beginning of 2023.
• Premium SSD v2 disks, our new general-purpose block storage tailored for SQL, NoSQL databases, and SAP grew capacity by 100 times.
• The total transactions of Premium Files, our file offering for Azure Virtual Desktop (AVD) and SAP, grew by more than 100% year over year (YoY).

In 2023, advancements and investments were made in five strategic areas aligning with customer workload patterns, cloud computing trends, and the evolution of AI.

Focused innovations for new workloads

We’ve made advancements in offering end-to-end solutions with unique storage capabilities, tailored to help customers bring new workloads to Azure, without the need to retrofit them.

As customers modernize applications with Kubernetes, we have observed customers’ workloads moving from “stateless” to “stateful” workloads. Azure Container Storage is custom designed to meet the needs of hosting stateful workloads where storage is tightly integrated with containers and offers outstanding data management, lifecycle management, price-performance, and scale. Azure Container Storage is the industry’s first platform-managed container-native storage service in the public cloud. It offers a path to manage persistent volumes and the backend storage options completely via the Kubernetes native experience.

Microsoft’s Azure Elastic SAN is a fully managed storage area network (SAN) service that simplifies deploying, scaling, managing, and configuring a SAN in the cloud. It is designed to address customers’ challenges in service management when it comes to migrating a large-scale SAN appliance to cloud. We are the first in the market in providing a fully managed SAN offering on cloud, with built-in cloud scale and high availability options. It adopts a SAN-like hierarchy, provisioning resources at the top-level resource, dynamically sharing resources across the workloads, managing security policies at the workload level. In addition, it enables customers to bring new data intensive workloads to Azure VMware Solution using Elastic SAN to serve the storage needs. Elastic SAN and Azure Container Storage, in public preview, showcase a paradigm shift by simplifying service management, purpose built for cloud native storage scenarios. General availability is planned for the upcoming months.

AI is leading a fresh wave of innovation powered by data. Azure Blob Storage stands at the forefront of this data explosion, providing excellent performance and high bandwidth access for all AI training needs. Azure Premium Block Blob, generally available since 2019, provides extremely low latency at competitive pricing for AI and machine learning (ML), IoT and streaming analytics, and interactive workloads. We also released Azure Managed Lustre in July which offers a fully managed distributed parallel file system tailored for high performance computing (HPC) and AI training workloads.

OneLake (which is the storage layer for Microsoft Fabric) is powered by the scale, performance, and rich capabilities of Azure Data Lake Storage (ADLS). ADLS has been a game changer for analytics with differentiated storage features including native file system capabilities that enable atomic meta data operations, resulting in a significant performance boost, especially when used with Azure Premium Block Blob. Multi-protocol access is another unique capability that allows customers to work with their data using a variety of access protocols (including REST, SFTP and NFS v3) eliminating data silos, costly data movement, and duplication.

Optimizations for mission critical workloads

We continue to add differentiated capabilities to enable enterprises to optimize their mission critical workloads hosted on Azure.  

With the general availability of Premium SSD v2, we have been actively onboarding solutions focused on mission critical data. Gigaom highlighted the superior performance and reduced total cost of ownership of SQL Server 2019 running on Azure Virtual Machines (VMs) with Premium SSD v2. Similar benefits apply to other database workloads like PostgreSQL. We will continue enhancing Premium SSD v2 capabilities and expand regional coverage in the upcoming year.

Azure Blob Storage powers data-intensive mission critical workloads globally, including some of the largest AI model training and inferencing. Cloud-native companies host exabytes of consumer-generated data and rely on the platform’s scale-out capabilities. Major retailers store trillions of user activities to power recommendation engines and inventory systems. The Cold storage tier, released in 2023, enables cost-efficient retention of this data.

Enterprises migrate Windows Server workloads to the cloud for modernization, supported by enhanced Azure Files and Azure File Sync capabilities. The preview of Microsoft Entra ID (previously named as Azure Active Directory support for Azure Files REST API) enables modern applications to use identity-based authentication and authorization for accessing SMB file shares. Azure Files offers comprehensive authentication support, including on-premises AD and Microsoft Entra ID for both Kerberos and REST. Unique to Azure, we also offer Microsoft Entra ID Kerberos authentication enabling seamless migration from on-premises Windows servers to Azure which keeps your data and permissions intact while eliminating complex domain join setups in the cloud. Enterprises migrating mission-critical applications such as SAP and Oracle rely on Azure NetApp Files (ANF) to meet their performance and data protection requirements.  In strategic partnership with NetApp, we introduced the public preview of ANF large volumes up to 500 TiB, addressing the needs of workloads that require larger capacity under a single namespace, such as HPC in Electronic Design Automation and Oil and Gas related applications, and the public preview of Cool Access, improving the TCO for infrequently accessed data.

Efficient data migration at scale is critical for successful cloud workload migration. We offer a comprehensive suite of migration solutions. Azure Storage Mover, a fully managed data migration service, improves the overall migration experience of large data sets. Azure File Sync allows hybrid synchronization between on-premises Windows Servers and Azure Files. AzCopy, a command-line utility, enables quick script-based migration jobs, while Azure Databox facilitates offline data transfer.

Expanding partner ecosystem

We transition customer workloads to Azure in collaboration with our partners, who complement our platform capabilities with innovative solutions. We have fostered numerous successful strategic partnerships and are delighted to see the partner community investing in exclusive Azure solutions.

Qumulo announced the next version of Azure Native Qumulo Scalable File Service, exclusive to Azure and built to leverage Azure Blob Storage’s scale and cost efficiency. This service supports exabytes of data under a single namespace, surpassing other public cloud file solutions, and facilitates high-scale enterprise network attached storage (NAS) data migration to Azure.

Commvault unveiled a purpose-built cyber resilience solution, exclusively on Azure. It empowers users to predict threats faster, achieve clean recoveries, and accelerate response times. It integrates seamlessly with Microsoft Azure OpenAI Service.

Since the launch of Pure Storage Cloud Block Storage offering on Azure in 2021, we have partnered closely with Pure Storage. They use unique Azure Storage features such as shared disks and were early adopters of Ultra and Premium SSD v2 disk storage. Our collaboration has expanded to containers through Portworx and the Azure VMware Solution.

Besides storage and backup partners, we collaborate with Atempo, Data Dynamics, Komprise, and Cirrus Data to deliver storage migration solutions, providing access to industry-leading file and block solutions to migrate petabytes of NAS and storage area network (SAN) data to Azure at no additional charge.

Industry contributions

We actively participate in industry organizations such as Storage Networking Industry Association (SNIA) and the Open Compute Platform (OCP), collaborating with industry leaders to share insights, influence standards, and contribute to the development of innovative solutions and best practices.

At the 2023 Storage Developer Conference, we presented Massively Scalable Storage for Stateful Containers on Azure, emphasizing the synergy between Azure Container Storage and Elastic SAN. This combination offers unified volume management across diverse storage backends and delivers a highly available block storage solution designed to scale to millions of input/output operations per second (IOPS) with fast pod attach and detach experience.

At Flash Memory Summit (FMS) conference we chartered the course for the future of flash as a HDD displacement technology in alignment with Azure Storage long-range plans.

We are committed to sustainability, striving for net carbon-zero goals in the next decade. Through the OCP Sustainability Project, we provide an open framework for the datacenter industry to adopt best practices for reusability and circularity and lead in the steering committee to drive progress.

Unparalleled commitment to quality

Our primary focus is on delivering a robust foundation for our storage services, adapting to evolving workloads like the emergence of AI and ML. This ongoing commitment covers reliability, durability, scalability, and performance, for our storage solutions while maintaining a low total cost of ownership (TCO) for Azure customers.

We assure customers of high data durability, backed by substantial investments in infrastructure, hardware, software, and streamlined processes. Our Zone-redundant Storage (ZRS) ensures data durability of at least 99.9999999999% (12 9’s) over a given year—the highest standard among the major cloud service providers (CSPs). Notably, we are the sole cloud provider among the major CSPs maintaining a 0% Annual Failure Rate (AFR) with our block storage offerings since launch.

Our unique ZRS offering provides a simple way to establish highly available solutions across three zones at a fraction of the costs. You can deploy a cross-zonal AKS cluster with persistent volumes hosted on ZRS disks to ensure data durability and availability during zonal outages. Clustered applications like SQL failover cluster instances (FCI) leveraging Windows Server Failover Cluster (WSFC) can also benefit from ZRS with high resiliency provided out of the box.

In a concerted effort to elevate customer support experience, we’ve integrated with Azure Copilot, empowering support engineers to troubleshoot more efficiently while delivering quality responses quickly. This has resulted in a noteworthy increase in customer satisfaction scores.

In closing, 2023 has been a year of profound learning and substantial progress. Explore our feature releases in Azure Updates. Our commitment is to empower you with Azure Storage innovations and enable the seamless execution of a diverse range of workloads. We invite you to continue trusting us as we anticipate exciting developments in 2024!

Source: microsoft.com