Enhanced Azure Arc integration with Datadog simplifies hybrid and multicloud observability

Businesses today are managing complex, distributed environments and need a ubiquitous computing platform for all workloads that can meet them where they are. We’ve seen an increasing need for customers to not only deploy, manage, and operate across on-premises and one or more clouds, but also to have better visibility and insights across all IT investments spanning cloud to edge.

Today, we’re delivering improved observability and management with the general availability of our enhanced Microsoft Azure Arc integration with Datadog. Building on our established collaboration, we are natively integrating Datadog with Azure Arc to meet customers where they are and provide rich insights from Azure Arc–enabled resources directly into Datadog dashboards. Customers can monitor real-time data during cloud migrations and performance of applications running both in the public cloud and in hybrid or multicloud environments.

Benefits of Azure Arc integration with Datadog

With the Azure Arc integration with Datadog, customers can:

◉ Monitor the connection status and agent version of Azure Arc–enabled servers, wherever they are running.

◉ Automatically add Azure tags to associated hosts in Datadog for additional context.

◉ Identify which Azure Arc–enabled servers have the Datadog Agent installed.

◉ Deploy the Datadog Agent onto your Azure Arc–enabled servers as an extension.

◉ Get unified billing for the Datadog service through Azure subscription invoicing.

Datadog is a cloud-scale monitoring and security platform for large-scale applications that aggregates data across your entire stack with more than 600 integrations for centralized visibility and faster troubleshooting on dynamic architectures. This provides developers and operations teams observability into every layer of their applications on Azure, so they diagnose performance issues quickly.

When Datadog first became an Azure Native ISV Service, it allowed customers to streamline their experience for purchasing, configuring, and managing Datadog directly inside the Azure portal. It reduced the learning curve for using Datadog to monitor the health and performance of your applications in Azure and sets customers up for a successful cloud migration or modernization.

For many customers, hybrid deployments are a durable and long-term strategy due to factors such as latency and compliance requirements, and we are committed to meeting customers wherever they are. With Azure Arc, we provide a consistent set of tools and services for customers to extend cloud technology across your distributed infrastructure. More than 12,000 customers are using Azure Arc, double the number a year ago. By partnering with organizations like Datadog, we are unlocking even more innovation and bringing Azure services into the tools our customers are already using.

Enhanced Azure Arc integration features

Features available with today’s general availability include:

Monitor the Arc connection status and agent version

Customers can easily identify any Azure Arc–enabled resources that are not in a connected state. You can also set up Datadog monitors to alert you immediately if the connection is unhealthy. Before this new integration, Azure Arc resources would look like any other virtual machine on-premises or in Azure. Now, you can access critical metadata to ensure your Azure Arc–enabled Windows and Linux servers, SQL servers, and Kubernetes clusters are secured and connected. IT operators will be able to troubleshoot much faster if a resource is disconnected and can quickly restore the connectivity to Azure Arc.

Datadog can also show which hosts are running an older version of Azure Arc. It then becomes easy to update the agent using Azure Update Management and utilize Azure Automation for latest updates to the Azure Arc agent whenever there is a new version.

Automatically add Azure tags for easy management and compliance tracking

A popular benefit of Azure Arc is using tags in Azure Resource Manager. Many organizations tag on-premises resources by cost center or datacenter server groups that are subject to specific regulations or requirements. Tags also create an audit trail to help trace the history of a particular resource and identify potential security issues when performing audits.

With the Azure Arc integration, Datadog can build rich visualizations and actionable alerts using the tags you have already created for Azure Arc–enabled resources. Now, when you perform patching or updates for Azure Arc–enabled servers, you get much richer insights to help validate software patches and troubleshoot application issues.

Easily identify which Azure Arc–enabled servers have the Datadog Agent

Azure Arc brings your hybrid and multicloud servers, Kubernetes clusters, and data services into a single dashboard for seamless management between environments. Aside from grouping resources with Azure Resource Manager, Azure Arc–enabled resources benefit from Azure role-based access control (RBAC), so different IT and developer teams can easily delegate access to their applications. For a centralized IT monitoring team, you can now ensure your Azure Arc–enabled resources have the Datadog Agent by cross-referencing these servers with agent data to get a real-time view of which Arc resources have Datadog Agent reporting.

Source: microsoft.com

Continue reading

Digital transformation for manufacturers requires additional IT/OT security

While every industry is vulnerable to a ransomware attack, manufacturers are at a particular risk. While digitization and automation have helped transform the industry, it has simultaneously opened up new attack vectors within organizations. Now the most targeted industry, the manufacturing industry, has seen a 300 percent increase in cyberattacks in a single year.

Beyond the tremendous growth in attacks, manufacturing companies make an ideal target for hackers due to the high value of the companies themselves, the high costs of unplanned downtime, and the highly visible impact that downtime has on consumers’ daily lives. With the risks so high, an enterprise-level solution that provides visibility and protection like Microsoft Defender for IoT is essential.

Visibility is the first step to network protection

The number of connected industrial control system (ICS)/operational technology (OT) devices in manufacturing facilities continues to grow. The benefits for the operations side of the house are clear, but the lack of visibility into them poses serious security risks for chief information security officers (CISOs).

Manufacturers often have no way to identify and monitor what all their connected devices are doing and with whom or what they are communicating. Worse, all too often they lack even a simple inventory of all the connected devices they have in their facilities. In case of an attack, the lack of visibility means that they have no way of tracing the attack vector the hacker took, making them vulnerable to a second wave and delaying recovery and remediation.

Continuous monitoring without impacting productivity

Microsoft Defender for IoT not only creates asset maps within minutes of being turned on, but it also provides continuous monitoring of every device in every facility around the world. Microsoft’s Section 52 has access to tens of trillions of identity, endpoint, and other signals each day. The threat intelligence from this specialized IoT and ICS research team produces high-impact insights that help keep manufacturers safe from attacks.

The agentless nature of the system protects companies without impacting production, no matter the topology of the network or the regulations governing the industry. And, with round-the-clock protection, Microsoft Defender for IoT can alert the SecOps team about an intrusion any time, any place.

Security for networks in an age of IT and OT convergence

As their digital transformations have progressed, manufacturers have seen their IT and OT environments converge. The air gap between them that ensured production would continue even if IT assets were taken offline is increasingly a thing of the past. With these trendlines, forward-thinking CISOs and their boards are taking proactive steps to protect the entire company from cyber-physical attacks that could have huge costs to safety, production, reputation, and the bottom line.

Fortunately, Microsoft Defender for IoT can usually be deployed in less than a single day per facility and works right out of the box for large enterprises and small, niche facilities. With it, defenders of OT networks have a powerful new tool at their disposal to help keep hackers out and people, production, and profits safe.

Source: microsoft.com

Continue reading

Optimize your Azure workloads with Azure Advisor Score

Modern engineering practices, like Agile and DevOps, are redirecting the ownership of security, operations, and cost management from centralized teams to workload owners—catalyzing innovations at a higher velocity than in traditional data centers. In this new world, workload owners are expected to build, deploy, and manage cloud workloads that are secure, reliable, performant, and cost-effective. If you’re a workload owner, you want well-architected deployments, so you might be wondering, how well are you doing today? Of all the actions you can take, which ones will make the biggest difference for your Azure workloads? And how will you know if you’re making progress?

That’s why we created Azure Advisor Score—to help you understand how well your Azure workloads are following best practices, assess how much you stand to gain by remediating issues, and prioritize the most impactful recommendations you can take to optimize your deployments.

Introducing Advisor Score

Advisor Score enables you to get the most out of your Azure investment using a centralized dashboard to monitor and work towards optimizing the cost, security, reliability, operational excellence, and performance of your Azure resources.

Advisor Score will help you:

◉ Assess how well you’re following the best practices defined by Azure Advisor and the Microsoft Azure Well-Architected Framework.

◉ Optimize your deployments by taking the most impactful actions first.

◉ Report on your well-architected progress over time.

Baselining is one great use case we’ve already seen with customers. You can use Advisor Score to baseline yourself and track your progress over time toward your goals by reviewing your score’s daily, weekly, or monthly trends. Then, to reach your goals, you can take action first on the individual recommendations and resources with the most impact.

How Advisor Score works

Advisor Score measures how well you’re adopting Azure best practices, comparing and quantifying the impact of the Advisor recommendations you’re already following, and the ones you haven’t implemented yet. Think of it as a gap analysis for your deployed Azure workloads.

The overall score is calculated on a scale from 0 percent to 100 percent both in aggregate and separately for cost, security (coming soon), reliability, operational excellence, and performance. A score of 100 percent means all your resources follow all the best practices recommended in Advisor. On the other end of the spectrum, a score of zero percent means that none of your resources follow the recommended best practices. Advisor Score weighs all resources, both those with and without active recommendations, by their individual cost relative to your total spend. This builds on the assumption that the resources which consume a greater share of your total investment in Azure are more critical to your workloads. Advisor Score also adds weight to resources with longstanding recommendations. The idea is that the accumulated impact of these recommendations grows the longer they go unaddressed.

Source: microsoft.com

Continue reading

Six reasons customers trust Azure to run their SAP solutions

As global organizations across every industry adjust to the new normal, SAP solutions are playing an increasingly vital role in addressing immediate needs and paving a path to a resilient future. Now more than ever, companies are realizing the value of running their SAP solutions in the cloud. While some are using advanced analytics to process their SAP data to make real-time business decisions, others are integrating their SAP and non-SAP data to build stronger supply chains. Whether it’s meeting urgent customer needs, empowering employees to make quick decisions, or planning for the future, customers running SAP solutions in the cloud have been well prepared to face the new reality. Check out how Walgreens delivers superior customer service with SAP solutions on Microsoft Azure.

Many organizations running their SAP solutions on-premises have become increasingly aware of the need to be more agile and responsive to real-time business needs. According to an IDC survey, 54 percent of enterprises expect the future demand for cloud software will increase. As global organizations seek agility, cost savings, risk reduction, and immediate insights from their ERP solutions, here are some reasons many of the largest enterprises choose Microsoft Azure as their trusted partner when moving their SAP solutions to the cloud.

1. Running SAP solutions on Azure delivers immediate insights and increased agility

Organizations running SAP solutions on Azure gain real-time and predictive insights that empower them to break into new ways of doing business. Azure offers the ability to tap into more than 100 cloud services, access SAP Cloud Platform, apply intelligent analytics, and also integrate with an organization’s existing productivity and collaboration tools such as Microsoft 365, Microsoft Teams, Microsoft Power Apps, and Microsoft Power BI.

With Azure, organizations can integrate their SAP and non-SAP data through an extensive portfolio of Azure data services and create real-time dashboard views of the current operations using SAP and Microsoft business intelligence tools. Using intelligent analytics deepens real-time and predictive insights to improve decision-making by responding dynamically as business conditions change, and how that change impacts your customers or products. Integration with Teams and Microsoft 365 improves team collaboration and enhances user experience and productivity. Using Microsoft Power Automate, Power Apps, and Power BI, organizations can create customized workflows, apps, and business insight reports without having to write any code.

2. An ever-evolving and growing set of Azure cloud services drives continuous innovation

While Zuellig Pharma is building an app that uses Azure blockchain services and data from the SAP Business Suite on HANA to track and capture counterfeit products and illegal parallel imports in its region, Walgreens plans to use AI and machine learning to develop new customer offerings quickly and respond in real time to changes in the marketplace.

Customers such as Rio Tinto are using Azure’s secure and scalable IoT applications to pilot a solution to take real-time data from trucks, drills, smelters, and other equipment and analyze it to gain equipment health, preemptive maintenance, supply chain efficiency, and other operational intelligence. Additionally, with DevOps with GitHub and Azure Kubernetes Service, customers can build, manage, and deploy applications on a massive global network.

3. Running SAP solutions on Azure offers costs savings

A Forrester study showed customers achieved more than 100 percent ROI, a 50 percent reduction in data center costs, and a 100 percent reduction in SAP release delays by migrating their SAP systems to Azure. Moving to Azure not only eliminates capital expenditure and cost of underutilized hardware, but it also offers cost management tools such as on-demand scaling during peak usage periods, using cheaper storage, and optimizing disaster recovery environments.

By running SAP solutions on Azure, organizations replace expensive, manual, and error-prone processes with automated, flexible processes, and with a single ticket-to-solution experience, enterprises empower employees to focus on value-added activities by putting data in their hands.

4. Running SAP solutions on Azure offers immense flexibility and scalability

Customers across every industry run their largest production SAP landscapes on Azure because it is a proven cloud platform certified by SAP to run their most mission-critical SAP applications. Azure offers the industry’s most performant and scale-able cloud infrastructure—offering 192 GB to 12 TB SAP HANA certified VMs in more regions than any other public cloud provider along with support for both Linux and Windows OS. Azure offers on-demand scalability and agility that reduces the time to market —customers can spin up or spin down resources as needed. For instance, Daimler AG reduced operational costs by 50 percent and increased agility by spinning up resources on-demand in 30 minutes with SAP S/4HANA and Azure. 

Azure also offers access to more than 1,000 pre-built integrations, out-of-the-box business services, SAP HANA services, and apps built by SAP and our partners. Customers such as Tate and Lyle appreciate that with Azure, they get access to compute, network, and storage resources preconfigured for SAP HANA that they didn’t have to build, install, or manage.

5. SAP solutions on Azure offer best-in-class security, compliance, and business continuity

Azure’s intelligent security services are backed by a $1 billion annual investment in enterprise-grade security and compliance offers and 3,500 cybersecurity professionals. Azure has the most compliance offerings of any public cloud. Azure offers the best-in-class security services such as Azure Sentinel for SIEM, Azure security center for threat monitoring, and Azure Active Directory for identity management. Additionally, customers can leverage built-in availability and recovery options such as Azure Backup and Azure Site Recovery to ensure business continuity and data protection. Microsoft teams work closely with partners to ensure that critical systems remain online during migration and offer a robust set of joint planning workshops, migration programs such as FastTrack, POCs, and training and certifications.

6. Organizations benefit from the trusted partnership between SAP and Microsoft

After decades of working together to serve our customers, SAP and Microsoft deepened their relationship by signing the Embrace initiative. As part of Embrace, SAP will lead with Azure to move on-premise SAP ERP and SAP S/4HANA customers to the cloud through industry-specific best practices, reference architectures, and cloud-delivered services. Our engineering teams co-residing in Germany and Redmond, Washington work together to develop joint reference architectures, product integration roadmaps, and best practices; our industry teams are jointly developing industry-specific transformation roadmaps, and our support teams have developed collaborative support models.

SAP and Microsoft have been partners for more than 25 years and are also mutual customers. Microsoft is the only cloud provider that’s been running SAP for its own finance, HR, and supply chains for the last 20 years, including SAP S/4HANA. Likewise, SAP has chosen Azure to run a growing number of its own internal system landscapes, including those based on SAP S/4HANA. Microsoft IT and SAP IT generously share their learnings from running SAP solutions on Azure with our customers.

More than 95 percent of Fortune 500 companies run their business on Azure. Our experience and history give us a powerful understanding of the needs of enterprise customers. Together with SAP, customers have trusted us with their most critical workloads for decades because we understand what it takes to support our customers in their journey to the cloud.

Source: microsoft.com

Continue reading

Azure Container Registry: Securing container workflows

Securing any environment requires multiple lines of defense. Azure Container Registry recently announced the general availability of features like Azure Private Link, customer-managed keys, dedicated data-endpoints, and Azure Policy definitions. These features provide tools to secure Azure Container Registry as part of the container end-to-end workflow.

Customer-managed keys

By default, when you store images and other artifacts in an Azure Container Registry, content is automatically encrypted at rest with Microsoft-managed keys.

Choosing Microsoft-managed keys means that Microsoft oversees managing the key’s lifecycle. Many organizations have stricter compliance needs, requiring ownership and management of the key’s lifecycle and access policies. In such cases, customers can choose customer-managed keys that are created and maintained in a customer’s Azure Key Vault instance. Since the keys are stored in Key Vault, customers can also closely monitor the access of these keys using the built-in diagnostics and audit logging capabilities  in Key Vault. Customer-managed keys supplement the default encryption capability with an additional encryption layer using keys provided by customers.

Private links

Container Registry previously had the ability to restrict access using firewall rules. With the introduction of Private Link, the registry endpoints are assigned private IP addresses, routing traffic within your virtual network and the service through a Microsoft backbone network.

Private Link support has been one of the top asks, allowing customers to benefit from the Azure management of their registry while benefiting from tightly controlled network ingress and egress.

Private links are available across a wide range of Azure resources with more coming soon, allowing a wide range of container workloads with the security of a private virtual network.

Dedicated data-endpoints

Private Link is the most secure way to control network access between clients and the registry as network traffic is limited to the Azure Virtual Network. When Private Link can't be used, dedicated data-endpoints can minimize data exfiltration concerns. Enabling dedicated data endpoints means they can configure firewall rules with fully qualified domain names ([registry].[region].data.azurecr.io) rather than a rule with wildcard (*.blob.core.windows.net) for all storage accounts.

You can enable dedicated data-endpoints using the Azure portal or the Microsoft Azure CLI. The data endpoints follow a regional pattern, <registry-name>.<region>.data.azurecr.io. In a geo-replicated registry, enabling data endpoints allows endpoints in all replica regions.

Azure built-in policies

Having security capabilities will secure your workflows if they’re implemented. To assure your Azure resources are following the best security practices, Azure Container Registry has added built-in Azure Policy definitions that you can leverage to enforce security rules. Here are some of the built-in policies that you can enable for your container registry:

◉ Container Registries should be encrypted with a customer-managed key. Audit Container Registries that do not have encryption enabled with customer-managed keys.

◉ Container Registries should not allow unrestricted network access. Audit Container Registries that do not have any network (IP or VNET) rules configured and allow all network access by default. Container Registries with at least one IP or firewall rule, or configured virtual network will be deemed compliant.

◉ Container Registries should use private links. Audit Container Registries that do not have at least one approved private endpoint connection. Clients in a virtual network can securely access resources that have private endpoint connections through private links.

Using Azure Policy, you can ensure that your registries stay compliant with your organization's compliance needs.

Source: microsoft.com

Continue reading

Advancing a culture of reliability at the pace of Azure

Like engineering reliability, Azure culture must balance the speed of the new with the stability of the known in the face of tremendous growth and unknowns. New hires bring new ideas and perspectives while veterans bring experience and institutional knowledge. Both contribute to the team culture, which defines how quality and innovation are valued and implemented.

To evolve the best quality outcomes, the Azure engineering team culture must be a place where ideas are openly shared, rigorously challenged, and effectively implemented. It’s a space where ideation and creativity thrive.

Skills, processes, and frameworks can be taught. But can culture be taught? How do you onboard new hires into a culture that values reliability?

Like so much about Azure, onboarding individuals and developing team culture at the speed of Azure has been fraught with challenges, and rich in learnings.

Onboarding engineers—Azure Engineering Boot Camp

The astounding growth rate of cloud computing has created an unprecedented demand for engineering roles worldwide. Within the Cloud + AI team, this results in hundreds of new hires joining Azure engineering teams each week. Hundreds of people to train on crucial skills, internal tools, and best practices. And hundreds of people to experience their first exposure to culture. How do you preserve institutional knowledge and disseminate culture that values reliability when waves of new people are onboarding constantly?

When a new engineer joins Microsoft, they spend their first day at New Employee Orientation (NEO), their first week getting familiar with their team and the environment, and their second week in the Azure Engineering Boot Camp (ABC).

ABC students delve into hands-on labs, learn Azure tools and services, and participate in lectures and activities to explore the engineering and business strategy. Many reliability principles are explicitly taught, including systems thinking, adaptive leadership, valuing diversity and inclusion, and customer development. For example, the “Systems Thinking” session looks at the tensions between feature velocity and quality, exploring the impact of whether and when adding more developers to a problem enhances feature velocity without compromising quality. But it is the way that these are presented that enables the principles to stick. The immersive week of in-person training enables trainers to model, coach, and help students develop reliability culture attributes along with technical skills in real-time.

A cohort of new engineers at a pre-COVID-19 ABC event.

While it’s not written anywhere in the agenda, ABC instructors place a high value on teaching culture. They understand that Microsoft needs new hires to step into their teams with more than technical skills. New hires need both the confidence to question the status quo, and a caring commitment to the team’s cultural dynamic. So that participants develop this expectation when they first arrive, ABC creates a shared experience by encouraging belonging and valuing diversity, and by inviting participants to challenge the curriculum and share personal experiences that may differ from those presented. ABC is a special place, creating a cohort of learners able to explore themselves in the context of the culture. Participants examine their authentic selves and find balance as they step into Azure.

Like the rest of Microsoft Azure, ABC is maturing and adapting to learner needs to improve its relevance as Azure scales. To offer a deeper sense of the program, here are three problems and the current thinking around them.

Problem 1: Enabling global parity

Some of the tenets that have made ABC in Redmond successful are a challenge for the global organization to replicate. Yet thinking holistically and inclusively are core principles of reliability engineering. In Redmond, ABC’s instructor-led, engaged cohort environment has been an essential aspect of enabling a culture of reliability for new hires, but is culture in Redmond the same as culture in Europe? In India? How can ABC hold to a global standard and achieve local relevance?

The team explored video options, but soon realized that Redmond-built video training would have opposite the desired effect on creating a vibrant engineering culture. Instead, the team worked with local stakeholders to set up regional training in the major development centers. The result is a ‘crawl, walk, run’ approach in which the regional centers could mature their Azure engineering education program at their own pace. The resulting global improvements to ABC are multiple, including an improved lab experience, more accurate and timely technical curriculum updates, and the diversity of contributors improves reliability outcomes for all.

One example is datacenter employee participation in ABC. This group tends to be early in career and/or from community colleges, with expertise in infrastructure engineering and operations as opposed to software engineering. The datacenter staff see ABC as a rare opportunity to connect with HQ teams, and they bring a real working experience to the cohort, most of whom won’t yet have had the chance to visit a datacenter. Having these groups learn together and share real-world datacenter experiences along with HQ insights helps both audiences grow.

Crawl, walk, run levels

At the ‘crawl’ level, events happen quarterly. A regional center hosts the ABC trainers and offers keynote speakers from the local leadership team, lunchtime lectures by local engineering teams describing their latest features and services, and a cohort of local veteran engineers to mentor students through capstone architecture design projects. The upside is that students are steeped in a mix of both local and global Azure culture during an event. The downside is that it happens only during ABC events, which are infrequent. An example deployment is the India Development Center in Hyderabad.

At the ‘walk’ level, a local program manager (PM) oversees a cohort of volunteer trainers to teach sessions and to seed the local flavor of Azure engineering culture. The local PM recruits keynote speakers, lecturers, and mentors. They manage communications, related communities, marketing to deepen the connections between new hires and the established engineering team. The benefit is that students have a deeper exposure to local Azure culture and a deeper relationship with local leadership. The PM’s work extends cultural exposure beyond ABC and invites alumni into more events and participation. An example deployment is the Israel Development Center in Herzliya.

At the ‘run’ level, two local FTE trainers who report into the Redmond team teach ABC, engage local speakers and lecturers to regionalize the program, and serve as ambassadors for Azure Engineering Learning all up so that the region is fully engaged in training events beyond ABC. This enables more cross-pollination of culture, with learning programs woven deeply in the fabric of the team. Learning culture becomes less event-based and more of an organic part of each day. An example deployment is the Irish Development Center in Dublin.

Dublin ABC FTE trainers Paul Mooney and Jyoti Gupta ready for their next session in December 2019.

Admittedly, it’s a progressive journey with learnings and improvements possible at all levels. The regions continue to share approaches, incubate, and experiment to continue learning and growth.

Problem 2: Personalizing factory education

By intent, a boot camp is rooted in the utilitarian premise of industrial age education in which a large body of students are brought to a specific level of knowledge and skill. In the military context, boot camp enables people from a broad swath of backgrounds, fitness levels, and skills to achieve a standard baseline of learning and physical outcomes. While ideal for teaching factory workers or soldiers how to read an instruction manual, it fails at tailoring education for individual skills development and does little for building the constructivist thinking necessary in a culture of reliability. How can an intensive onboarding program provide a tailored learning experience at scale?

ABC is sensitive to this need, but resolution is a work in progress. The curriculum is intended to provide students with a survey of the tools, technologies, and best practices they will encounter early and often in their Azure career. It focuses on providing an essential overview, a brief practical experience, and self-paced resources for continued learning. After ABC, students are expected to define a personal course of learning that integrates additional training events on offer as well as self-paced resources.

Anecdotally we hear that ABC graduates are more confident about making suggestions and assertions earlier in role because they experienced that behavior being modeled and valued in the ABC classroom. Trainers use stories from real world student experiences to make reliability theory more concrete. ABC PM and Trainer Devidas Gupta comments, “The best learning is when I get to stand aside and enable students to share and discuss with each other with minimal guidance or facilitation from me. That’s when I know they’ll walk out of the learning experience and into their team with confidence and willingness to engage with others effectively.”

A two-speed audience

Student critiques of ABC fall roughly into two groups. Some early-in-career types feel the pace is too fast and doesn’t provide enough context and examples, while seasoned engineers say that the curriculum is too basic. They ask for more and deeper coverage of advanced topics.

On the surface, it would seem that dividing the program into basic and advanced cohorts with appropriate topics and pacing for each would be the best solution when resources and logistics can support it. Historically, ABC has taken a different approach to build a more resilient growth culture, as both groups benefit from learning together. Early in career engineers can learn from those more experienced, and seasoned professionals can get new perspectives and approaches from recent college graduates. The classroom becomes a crucible for team cultural experience.

The instructor-led format requires that instructors seek to understand the makeup of each cohort so that they can adjust the pace, add context, or explore the edges of the curriculum to match the learning experience to the cohort needs. Instructors come early, stay late, and often engage students after ABC completion to mentor them, connect them with subject matter experts (SMEs), or offer guidance and resources. This deep and ongoing engagement is what makes ABC so special.

Piloting new approaches

Recent customer developments are enabling large-scale hiring in the East Coast of the U.S. These new recruits mostly come from the early in career background and need a more DevOps-focused curriculum. This gives ABC the opportunity to slow the pace and change topic breadth while keeping culture central to the experience. The intent is to extend the learning time frame to offer the basics in greater context, then connect the cohort with more seasoned new hires to broaden the experience. First piloted this spring and fine-tuned over the summer, the curriculum is now in full rollout.

Problem 3: Creating culture at a distance during COVID-19

Current COVID-19 restrictions have made live instructor-led events impossible. In its current format, a video recording of ABC would be a much-diminished learning experience, and most of the cultural dynamic would be lost. Yet new recruits are joining each week and need to be onboarded and trained. How can we bring culture to new hires during this pandemic? How do we build new capabilities to prepare for the future?

Immediate need

ABC is working closely with the NEO team to make sure that new hires joining have the best possible onboarding experience. To build culture, ABC is exploring assigning learning mentors called “camp counselors” to work with small cohorts of new hires and guide them through available self-paced learning options that roughly align to ABC.

ABC FTE trainer, Tim Colbert, pilots a video-based course.

Short term solution

ABC has pivoted to a virtual video series for self-paced learning. The videos focus on the technical skills and demos and are supported by Microsoft Teams-based office hour discussions, hosted by instructors and topic SMEs. Camp counselors act as learning mentors to support the experience. This way, students have a live venue for connecting with mentors, instructors, and other students to garner familiarity with the culture and have some personalized support despite the lost classroom experience.

Long term approach

Based on student response, ABC will learn, adapt, and scale video and teams-based training approaches, sensitive to how they model, support, and grow Azure’s culture of reliability. ABC will continue to pilot new approaches and build more capacity for additional training experiences. Beyond ABC, reliability is covered in the “Implementing SLOs” course and several “Cloud Talks.”

Looking forward

The core of Azure reliability starts with its engineering team culture. Teams need to be a place where ideation is safe, dissention is explored, and passion for quality is foremost. Just as a high velocity of change can impede system reliability, the pace of new hires can disrupt team cultural development. It is critical for onboarding training to model and develop the desired cultural attributes, yet this becomes increasingly challenging at scale, whether that scale is global, across learning levels, or in the face of a pandemic. ABC training will continue to pilot new approaches and evolve better solutions to ensure all engineers can effectively participate in improving reliability from their earliest days in role.

Source: microsoft.com

Continue reading

Advancing the outage experience—automation, communication, and transparency

In the cloud industry, we have a commitment to bring our customers the latest technology at scale, keeping customers and our platform secure, and ensuring that our customer experience is always optimal. For this to happen Azure is subject to a significant amount of change—and in rare circumstances, it is this change that can bring about unintended impact for our customers. As previously mentioned in this series of blog posts we take change very seriously and ensure that we have a systematic and phased approach to implementing changes as carefully as possible.

We continue to identify the inherent (and sometimes subtle) imperfections in the complex ways that our architectural designs, operational processes, hardware issues, software flaws, and human factors can align to cause service incidents—also known as outages. The reality of our industry is that impact caused by change is an intrinsic problem. When we think about outage communications we tend not to think of our competition as being other cloud providers, but rather the on-premises environment. On-premises change windows are controlled by administrators. They choose the best time to invoke any change, manage and monitor the risks, and roll it back if failures are observed.

Similarly, when an outage occurs in an on-premises environment, customers and users feel that they are more ‘in the know.’ Leadership is promptly made fully aware of the outage, they get access to support for troubleshooting, and expect that their team or partner company would be in a position to provide a full Post Incident Report (PIR)—previously called Root Cause Analysis (RCA)—once the issue is understood. Although our data analysis supports the hypothesis that time to mitigate an incident is faster in the cloud than on-premises, cloud outages can feel more stressful for customers when it comes to understanding the issue and what they can do about it.

Introducing our communications principles

During cloud outages, some customers have historically reported feeling as though they’re not promptly informed, or that they miss necessary updates and therefore lack a full understanding of what happened and what is being done to prevent future issues occurring. Based on these perceptions, we now operate by five pillars that guide our communications strategy—all of which have influenced our Azure Service Health experience in the Azure portal and include:

1. Speed
2. Granularity
3. Discoverability
4. Parity
5. Transparency

Speed

We must notify impacted customers as quickly as possible. This is our key objective around outage communications. Our goal is to notify all impacted Azure subscriptions within 15 minutes of an outage. We know that we can’t achieve this with human beings alone. By the time an engineer is engaged to investigate a monitoring alert to confirm impact (let alone engaging the right engineers to mitigate it, in what can be a complicated array of interconnectivities including third-party dependencies) too much time has passed. Any delay in communications leaves customers asking, “Is it me or is it Azure?” Customers can then spend needless time troubleshooting their own environments. Conversely, if we decide to err on the side of caution and communicate every time we suspect any potential customer impact, our customers could receive too many false positives. More importantly, if they are having an issue with their own environment, they could easily attribute these unrelated issues to a false alarm being sent by the platform. It is critical that we make investments that enable our communications to be both fast and accurate.

Last month, we outlined our continued investment in advancing Azure service quality with artificial intelligence: AIOps. This includes working towards improving automatic detection, engagement, and mitigation of cloud outages. Elements of this broader AIOps program are already being used in production to notify customers of outages that may be impacting their resources. These automatic notifications represented more than half of our outage communications in the last quarter. For many Azure services, automatic notifications are being sent in less than 10 minutes to impacted customers via Service Health—to be accessed in the Azure portal, or to trigger Service Health alerts that have been configured, more on this below.

With our investment in this area already improving the customer experience, we will continue to expand the scenarios in which we can notify customers in less than 15 minutes from the impact start time, all without the need for humans to confirm customer impact. We are also in the early stages of expanding our use of AI-based operations to identify related impacted services automatically and, upon mitigation, send resolution communications (for supported scenarios) as quickly as possible.

Granularity

We understand that when an outage causes impact, customers need to understand exactly which of their resources are impacted. One of the key building blocks in getting the health of specific resources are Resource Health signals. The Resource Health signal will check if a resource, such as a virtual machine (VM), SQL database, or storage account, is in a healthy state. Customers can also create Resource Health alerts, which leverage Azure Monitor, to let the right people know if a particular resource is having issues, regardless of whether it is a platform-wide issue or not. This is important to note: a Resource Health alert can be triggered due to a resource becoming unhealthy (for example, if the VM is rebooted from within the guest) which is not necessarily related to a platform event, like an outage. Customers can see the associated Resource Health checks, arranged by resource type.

We are building on this technology to augment and correlate each customer resource(s) that has moved into an unhealthy state with platform outages, all within Service Health. We are also investigating how we can include the impacted resources in our communication payloads, so that customers won’t necessarily need to sign in to Service Health to understand the impacted resources—of course, everyone should be able to consume this programmatically.

All of this will allow customers with large numbers of resources to know more precisely which of their services are impacted due to an outage, without having to conduct an investigation on their side. More importantly, customers can build alerts and trigger responses to these resource health alerts using native integrations to Logic Apps and Azure Functions.

Discoverability

Although we support both ‘push’ and ‘pull’ approaches for outage communications, we encourage customers to configure relevant alerts, so the right information is automatically pushed out to the right people and systems. Our customers and partners should not have to go searching to see if the resources they care about are impacted by an outage—they should be able to consume the notifications we send (in the medium of their choice) and react to them as appropriate. Despite this, we constantly find that customers visit the Azure Status page to determine the health of services on Azure.

Before the introduction of the authenticated in-portal Service Health experience, the Status page was the only way to discover known platform issues. These days, this public Status page is only used to communicate widespread outages (for example, impacting multiple regions and/or multiple services) so customers looking for potential issues impacting them don’t see the full story here. Since we rollout platform changes as safely as possible, the vast majority of issues like outages only impact a very small ‘blast radius’ of customer subscriptions. For these incidents, which make up more than 95 percent of our incidents, we communicate directly to impacted customers in-portal via Service Health.

We also recently integrated the ‘Emerging Issues’ feature into Service Health. This means that if we have an incident on the public Status page, and we have yet to identify and communicate to impacted customers, users can see this same information in-portal through Service Health, thereby receiving all relevant information without having to visit the Status page. We are encouraging all Azure users to make Service Health their ‘one stop shop’ for information related to service incidents, so they can see issues impacting them, understand which of their subscriptions and resources are impacted, and avoid the risk of making a false correlation, such as when an incident is posted on the Status page, but is not impacting them.

Most importantly, since we’re talking about the discoverability principle, from within Service Health customers can create Service Health alerts, which are push notifications leveraging the integration with Azure Monitor. This way, customers and partners can configure relevant notifications based on who needs to receive them and how they would best be notified—including by email, SMS, LogicApp, and/or through a webhook that can be integrated into service management tools like ServiceNow, PagerDuty, or Ops Genie.

To get started with simple alerts, consider routing all notifications to email a single distribution list. To take it to the next level, consider configuring different service health alerts for different use cases—maybe all production issues notify ServiceNow, maybe dev and test or pre-production issues might just email the relevant developer team, maybe any issue with a certain subscription also sends a text message to key people. All of this is completely customizable, to ensure that the right people are notified in the right way.

Parity

All Azure users should know that Service Health is the one place to go, for all service impacting events. First, we ensure that this experience is consistent across all our different Azure Services, each using Service Health to communicate any issues. As simple as this sounds, we are still navigating through some unique scenarios that make this complex. For example, most people using Azure DevOps don’t interact with the Azure portal. Since DevOps does not have its own authenticated Service Health experience, we can’t communicate updates directly to impacted customers for small DevOps outages that don’t justify going to the public Status page. To support scenarios like this, we have stood up the Azure DevOps status page where smaller scale DevOps outages can be communicated directly to the DevOps community.

Second, the Service Health experience is designed to communicate all impacting events across Azure—this includes maintenance events as well as service or feature retirements, and includes both widespread outages and isolated hiccups that only impact a single subscription. It is imperative that for any impact (whether it is potential, actual or upcoming) customers can expect the same experience and put in place a predictable action plan across all of their services on Azure.

Lastly, we are working towards expanding our philosophy of this pillar to extend to other Microsoft cloud products. We acknowledge that, at times, navigating through our different cloud products such as Azure, Microsoft 365, and Power Platform can sometimes feel like navigating technologies from three different companies. As we look to the future, we are invested in harmonizing across these products to bring about a more consistent, best-in-class experience.

Transparency

As we have mentioned many times in the Advancing Reliability blog series, we know that trust is earned and needs to be maintained. When it comes to outages, we know that being transparent about what is happening, what we know, and what we don’t know is critically important. The cloud shouldn’t feel like a black box. During service issues, we provide regular communications to all impacted customers and partners. Often, in the early stages of investigating an issue, these updates might not seem detailed until we learn more about what’s happening. Even though we are committed to sharing tangible updates, we generally try to avoid sharing speculation, since we know customers make business decisions based on these updates during outages.

In addition, an outage is not over once customer impact is mitigated. We could still be learning about the complexities of what led to the issue, so sometimes the message sent at or after mitigation is a fairly rudimentary summation of what happened. For major incidents, we follow this up with a PIR generally within three days, once the contributing factors are better understood.

For incidents that may have impacted fewer subscriptions, our customers and partners can request more information from within Service Health by requesting a PIR for the incident. We have heard feedback in the past that PIRs should be even more transparent, so we continue to encourage our incident managers and communications managers to provide as much detail as possible—including information about the issue impact, and our next steps to mitigate future risk. Ideally to ensure that this class of issue is less likely and/or less impactful moving forward.

While our industry will never be completely immune to service outages, we do take every opportunity to look at what happened from a holistic perspective and share our learnings. One of the future areas of investment at which we are looking closely, is how best to keep customers updated with the progress we are making on the commitments outlined in our PIR next steps. By linking our internal repair items to our external commitments in our next steps, customers and partners will be able to track the progress that our engineering teams are making to ensure that corrective actions are completed.

Our communications across all of these scenarios (outages, maintenance, service retirements, and health advisories) will continue to evolve, as we learn more and continue investing in programs that support these five pillars.

Reliability is a shared responsibility

While Microsoft is responsible for the reliability of the Azure platform itself, our customers and partners are responsible for the reliability of their cloud applications—including using architectural best practices based on the requirements of each workload. Building a reliable application in the cloud is different from traditional application development. Historically, customers may have purchased levels of redundant higher-end hardware to minimize the chance of an entire application platform failing. In the cloud, we acknowledge up front that failures will happen. As outlined several times above, we will never be able to prevent all outages. In addition to Microsoft trying to prevent failures, when building reliable applications in the cloud your goal should be to minimize the effects of any single failing component.

To that end, we recently launched the Microsoft Azure Well-Architected Framework—a set of guiding tenets that can be used to improve the quality of a workload. Reliability is one of the five pillars of architectural excellence alongside Cost Optimization, Operational Excellence, Performance Efficiency, and Security. If you already have a workload running in Azure and would like to assess your alignment to best practices in one or more of these areas, try the Microsoft Azure Well-Architected Review.

Specifically, the Reliability pillar describes six steps for building a reliable Azure application. Define availability and recovery requirements based on decomposed workloads and business needs. Use architectural best practices to identify possible failure points in your proposed/existing architecture and determine how the application will respond to failure. Test with simulations and forced failovers to test both detection and recovery from various failures. Deploy the application consistently using reliable and repeatable processes. Monitor application health to detect failures, monitor indicators of potential failures, and gauge the health of your applications. Finally, respond to failures and disasters by determining how best to address it based on established strategies.

Returning to our core topic of outage communications, we are working to incorporate relevant Well-Architected guidance into our PIRs in the aftermath of each service incident. Customers running critical workloads will be able to learn about specific steps to improve reliability that would have helped to avoid and lessen impact from that particular outage. For example, if an outage only impacted resources within a single Availability Zone, we will call this out as part of the PIRs and encourage impacted customers to consider zonal redundancies for their critical workloads.

Going forward

We outlined how Azure approaches communications during and after service incidents like outages. We want to be transparent about our five communication pillars, to explain both our progress to date and the areas in which we’re continuing to invest. Just as our engineering teams endeavor to learn from each incident to improve the reliability of the platform, our communications teams endeavor to learn from each incident to be more transparent, to get customers and partners the right details to make informed decisions, and to support customers and partners as best as possible during each of these difficult situations.

We are confident that we are making the right investments to continuing improving in this space, but we are increasingly looking for feedback on whether our communications are hitting the mark. We include an Azure post-incident survey at the end of each PIR we publish. We strive to review every response to learn from our customers and partners and validate whether we are focusing on the right areas and to keep improving the experience.

We continue to identify the inherent (and sometimes subtle) imperfections in the complex ways that our architectural designs, operational processes, hardware issues, software flaws, and human factors align to cause outages. Since trust is earned and needs to be maintained, we are committed to being as transparent as possible—especially during these infrequent but inevitable service issues.

Source: microsoft.com

Continue reading

Advancing resilience through chaos engineering and fault injection

Developing large-scale, distributed applications has never been easier, but there is a catch. Yes, infrastructure is provided in minutes thanks to your public cloud, there are many language options to choose from, swaths of open source code available to leverage, and abundant components and services in the marketplace to build upon. Yes, there are good reference guides that help give a leg up on your solution architecture and design, such as the Azure Well-Architected Framework and other resources in the Azure Architecture Center. But while application development is easier, there’s also an increased risk of impact from dependency disruptions. However rare, outages beyond your control could occur at any time, your dependencies could have incidents, or your key services/systems could become slow to respond. Minor disruptions in one area can be magnified or have longstanding side effects in another. These service disruptions can rob developer productivity, negatively affect customer trust, cause lost business, and even impact an organization’s bottom line.

Modern applications, and the cloud platforms upon which they are built, need to be designed and continuously validated for failure. Developers need to account for known and unknown failure conditions, applications and services must be architected for redundancy, algorithms need retry and back-off mechanisms. Systems need to be resilient to the scenarios and conditions caused by infrequent but inevitable production outages and disruptions. This post is designed to get you thinking about how best to validate typical failure conditions, including examples of how we at Microsoft validate our own systems.

Resilience

Resilience is the ability of a system to fail gracefully in the face of—and eventually recover from—disruptive events. Validating that an application, service, or platform is resilient is equally as important as building for failure. It is easy and tempting to validate the reliability of individual components in isolation and infer that the entire system will be just as reliable, but that could be a mistake. Resilience is a property of an entire system, not just its components. To understand if a system is truly resilient, it is best to measure and understand the resilience of the entire system in the environment where it will run. But how do you do this, and where do you start?

Chaos engineering and fault injection

Chaos engineering is the practice of subjecting a system to the real-world failures and dependency disruptions it will face in production. Fault injection is the deliberate introduction of failure into a system in order to validate its robustness and error handling.

Through the use of fault injection and the application of chaos engineering practices generally, architects can build confidence in their designs – and developers can measure, understand, and improve the resilience of their applications. Similarly, Site Reliability Engineers (SREs) and in fact anyone who holds their wider teams accountable in this space can ensure that their service level objectives are within target, and monitor system health in production. Likewise, operations teams can validate new hardware and datacenters before rolling out for customer use. Incorporation of chaos techniques in release validation gives everyone, including management, confidence in the systems that their organization is building.

Throughout the development process, as you are hopefully doing already, test early and test often. As you prepare to take your application or service to production, follow normal testing practices by adding and running unit, functional, stress, and integration tests. Where it makes sense, add test coverage for failure cases, and use fault injection to confirm error handling and algorithm behavior. For even greater impact, and this is where chaos engineering really comes into play, augment end-to-end workloads (such as stress tests, performance benchmarks, or a synthetic workload) with fault injection. Start in a pre-production test environment before performing experiments in production, and understand how your solution behaves in a safe environment with a synthetic workload before introducing potential impact to real customer traffic.

Healthy use of fault injection in a validation process might include one or more of the following:

◉ Ad hoc validation of new features in a test environment:

A developer could stand up a test virtual machine (VM) and run new code in isolation. While executing existing functional or stress tests, faults could be injected to block network access to a remote dependency (such as SQL Server) to prove that the new code handles the scenario correctly.

◉ Automated fault injection coverage in a CI/CD pipeline, including deployment or resiliency gates:

Existing end-to-end scenario tests (such as integration or stress tests) can be augmented with fault injection. Simply insert a new step after normal execution to continue running or run again with some faults applied. The addition of faults can find issues that would normally not be found by the tests or to accelerate discovery of issues that might be found eventually.

◉ Incident fix validation and incident regression testing:

Fault injection can be used in conjunction with a workload or manual execution to induce the same conditions that caused an incident, enabling validation of a specific incident fix or regression testing of an incident scenario.

◉ BCDR drills in a pre-production environment:

Faults that cause database failover or take storage offline can be used in BCDR drills, to validate that systems behave appropriately in the face of these faults and that data is not lost during any failover tests.

◉ Game days in production:

A ‘game day’ is a coordinated simulation of an outage or incident, to validate that systems handle the event correctly. This typically includes validation of monitoring systems as well as human processes that come into play during an incident. Teams that perform game days can leverage fault injection tooling, to orchestrate faults that represent a hypothetical scenario in a controlled manner.

Typical release pipeline

This figure shows a typical release pipeline, and opportunities to include fault injection:

An investment in fault injection will be more successful if it is built upon a few foundational components:

◉ Coordinated deployment pipeline.
◉ Automated ARM deployments.
◉ Synthetic runners and synthetic end-to-end workloads.
◉ Monitoring, alerting, and livesite dashboards.

With these things in place, fault injection can be integrated in the deployment process with little to no additional overhead – and can be used to gate code flow on its way to production.

Localized rack power outages and equipment failures have been found as single points of failure in root cause analysis of past incidents. Learning that a service is impacted by, and not resilient to, one of these events in production is a timebound, painful, and expensive process for an on-call engineer. There are several opportunities to use fault injection to validate resilience to these failures throughout the release pipeline in a controlled environment and timeframe, which also gives more opportunity for the code author to lead an investigation of issues uncovered. A developer who has code changes or new code can create a test environment, deploy the code, and perform ad hoc experiments using functional tests and tools with faults that simulate taking dependencies offline – such as killing VMs, blocking access to services, or simply altering permissions. In a staging environment, injection of similar faults can be added to automated end-to-end and integration tests or other synthetic workloads. Test results and telemetry can then be used to determine impact of the faults and compared against baseline performance to block code flow if necessary.

In a pre-production or ‘Canary’ environment, automated runners can be used with faults that again block access to dependencies or take them offline. Monitoring, alerting, and livesite dashboards can then be used to validate that the outages were observed as well as that the system reacted and compensated for the issue—that it demonstrated resilience. In this same environment, SREs or operations teams may also perform business continuity/disaster recovery (BCDR) drills, using fault injection to take storage or databases offline and once again monitoring system metrics to validate resilience and data integrity. These same Canary activities can also be performed in production where there is real customer traffic, but doing so incurs a higher possibility of impact to customers so it is recommended only to do this after leveraging fault injection earlier in the pipeline. Establishing these practices and incorporating fault injection into a deployment pipeline allows systematic and controlled resilience validation which enables teams to mitigate issues, and improve application reliability, without impacting end customers.

Fault injection at Microsoft

At Microsoft, some teams incorporate fault injection early in their validation pipeline and automated test passes. Different teams run stress tests, performance benchmarks, or synthetic workloads in their automated validation gates as normal and a baseline is established. Then the workload is run again, this time with faults applied – such as CPU pressure, disk IO jitter, or network latency. Workload results are monitored, telemetry is scanned, crash dumps are checked, and Service Level Indicators (SLIs) are compared with Service Level Objectives (SLOs) to gauge the impact. If results are deemed a failure, code may not flow to the next stage in the pipeline.

Other Microsoft teams use fault injection in regular Business Continuity, Disaster Recovery (BCDR) drills, and Game Days. Some teams have monthly, quarterly, or half-yearly BCDR drills and use fault injection to induce a disaster and validate both the recovery process as well as the alerting, monitoring and live site processes. This is often done in a pre-production Canary environment before being used in production itself with real customer traffic. Some teams also carry out Game Days, where they come up with a hypothetical scenario, such as replication of a past incident, and use fault injection to help orchestrate it. Faults, in this case, might be more destructive—such as crashing VMs, turning off network access, causing database failover, or simulating an entire datacenter going offline. Again, normal live site monitoring and alerting are used, so your DevOps and incident management processes are also validated. To be kind to all involved, these activities are typically performed during business hours and not overnight or over a weekend.

Our operations teams also use fault injection to validate new hardware before it is deployed for customer use. Drills are performed where the power is shut off to a rack or datacenter, so the monitoring and backup systems can be observed to ensure they behave as expected.

At Microsoft, we use chaos engineering principles and fault injection techniques to increase resilience, and confidence, in the products we ship. They are used to validate the applications we deliver to customers, and the services we make available to developers. They are used to validate the underlying Azure platform itself, to test new hardware before it is deployed. Separately and together, these contribute to the overall reliability of the Azure platform—and improved quality in our services all up.

Unintended consequences

Remember, fault injection is a powerful tool and should be used with caution. Safeguards should be in place to ensure that faults introduced in a test or pre-production environment will not also affect production. The blast radius of a fault scenario should be contained to minimize impact to other components and to end customers. The ability to inject faults should have restricted access, to prevent accidents and prevent potential use by hackers with malicious intent. Fault injection can be used in production, but plan carefully, test first in pre-production, limit the blast radius, and have a failsafe to ensure that an experiment can be ended abruptly if needed. The 1986 Chernobyl nuclear accident is a sobering example of a fault injection drill gone wrong. Be careful to insulate your system from unintended consequences.

Chaos as a service?

This is an exciting space with so much potential to improve cloud service reliability and reduce the impact of rare but inevitable disruptions. There are many teams doing lots of interesting things in this space, and we’re exploring how best to bring all these disparate tools and faults together to make our lives easier—for our internal developers building Azure services, for built-on-Azure services like Microsoft 365, Microsoft Teams, and Dynamics, and eventually for our customers and partners to use the same tooling to wreak havoc on (and ultimately improve the resilience of) their own applications and solutions.

Source: microsoft.com

Continue reading

Azure Support API: Create and manage Azure support tickets programmatically

Large enterprise customers running business-critical workloads on Azure manage thousands of subscriptions and use automation for deployment and management of their Azure resources. Expert support for these customers is critical in achieving success and operational health of their business. Today, customers can keep running their Azure solutions smoothly with self-help resources, such as diagnosing and solving problems in the Azure portal, and by creating support tickets to work directly with technical support engineers.

We have heard feedback from our customers and partners that automating support procedures is key to help them move faster in the cloud and focus on their core business. Integrating internal monitoring applications and websites with Azure support tickets has been one of their top asks. Customers expect to create, view, and manage support tickets without having to sign-in to the Azure portal. This gives them the flexibility to associate the issues they are tracking with the support tickets they raise with Microsoft. The ability to programmatically raise and manage support tickets when an issue occurs is a critical step for them in Azure usability.

We’re happy to share that the Azure Support API is now generally available. With this API, customers can integrate the creation and management of support tickets directly into their IT service management (ITSM) system, and automate common procedures.

Using the Azure Support API, you can:

◉ Create a support ticket for technical, billing, subscription management, and subscription and service limits (quota) issues.

◉ Get a list of support tickets with detailed information, and filter by status or created date.

◉ Update severity, status, and contact information.

◉ Manage all communications for a support ticket.

Benefits of Azure Support API

Reduce the time between finding an issue and getting support from Microsoft

A typical troubleshooting process when the customer encounters an Azure issue looks something like this:

On step five, if the issue is unresolved and identified to be on the Azure side, customers navigate to the Azure portal, to contact support. With programmatic case management access, customers can automate their support process with their internal tooling to create and manage their support tickets, thus reducing the time between finding an issue and contacting support.

Customers now have one end-end process that goes smoothly from internal to external without the person filing the issue having to deal with the complexity and challenges between separate case management systems.

Create support tickets via ARM templates

Deploying an ARM template that creates resources can sometimes result in a ResourceQuotaExceeded deployment error, indicating that you have exceeded your Azure subscription and service limits (quotas). This happens because quotas are applied in the resource group, subscription, account, and other scopes. For example, your subscription may be configured to limit the number of cores for a region. If you attempt to deploy a virtual machine with more cores than the permitted amount, you receive an error stating the quota has been exceeded. The way to resolve it is to request a quota increase by filing a support ticket. With Support APIs in place, you can avoid signing in to the Azure portal to create a ticket, instead request quota increases directly via ARM templates.

Continue reading