Burst 4K encoding on Azure Kubernetes Service

Burst encoding in the cloud with Azure and Media Excel HERO platform.

Content creation has never been as in demand as it is today. Both professional and user-generated content has increased exponentially over the past years. This puts a lot of stress on media encoding and transcoding platforms. Add the upcoming 4K and even 8K to the mix and you need a platform that can scale with these variables. Azure Cloud compute offers a flexible way to grow with your needs. Microsoft offers various tools and products to fully support on-premises, hybrid, or native cloud workloads. Azure Stack offers support to a hybrid scenario for your computing needs and Azure ARC helps you to manage hybrid setups.

Finding a solution

Generally, 4K/UHD live encoding is done on dedicated hardware encoder units, which cannot be hosted in a public cloud like Azure. With such dedicated hardware units hosted on-premise that need to push 4K into the Azure data center the immediate problem we face is a need for high bandwidth network connection between the encoder unit on-premise and Azure data center. In general, it's a best practice to ingest into multiple regions, increasing the load on the network connected between the encoder and the Azure Datacenter.

How do we ingest 4K content reliably into the public cloud?

Alternatively, we can encode the content in the cloud. If we can run 4K/UHD live encoding in Azure, its output can be ingested into Azure Media Services over the intra-Azure network backbone which provides sufficient bandwidth and reliability.

How can we reliably run and scale 4K/UHD live encoding on the Azure cloud as a containerized solution? Let's explore below.

Azure Kubernetes Service

With Azure Kubernetes Services (AKS) Microsoft offers a managed Kubernetes platform to customers. It is a hosted Kubernetes platform without having to spend a lot of time creating a cluster with all the necessary configuration burden like networking, cluster masters, and OS patching of the cluster nodes. It also comes with pre-configured monitoring seamlessly integrating with Azure Monitor and Log Analytics. Of course, it still offers flexibility to integrate your own tools. Furthermore, it is still just the plain vanilla Kubernetes and as such is fully compatible with any existing tooling you might have running on any other standard Kubernetes platform.

Media Excel encoding

Media Excel is an encoding and transcoding vendor offering physical appliance and software-based encoding solutions. Media Excel has been partnering with Microsoft for many years and engaging in Azure media customer projects. They are also listed as recommended and tested contribution encoder for Azure Media Services for fMP4. There has also work done by both Media Excel and Microsoft to integrate SCTE-35 timed metadata from Media Excel encoder to an Azure Media Services Origin supporting Server-Side Ad Insertion (SSAI) workflows.

Networking challenge

With increasing picture quality like 4K and 8K, the burden on both compute and networking becomes a significant architecting challenge. In a recent engagement with a customer, we needed to architect a 4K live streaming platform with a challenge of limited bandwidth capacity from the customer premises to one of our Azure Datacenters. We worked with Media Excel to build a scalable containerized encoding platform on AKS. Utilizing cloud compute and minimizing network latency between Encoder and Azure Media Services Packager. Multiple bitrates with a top bitrate up to 4Kp60@20Mbps of the same source are generated in the cloud and ingested into the Azure Media Services platform for further processing. This includes Dynamic Encryption and Packaging. This setup enables the following benefits:

◉ Instant scale to multiple AKS nodes

◉ Eliminate network constraints between customer and Azure Datacenter

◉ Automated workflow for containers and easy separation of concern with container technology

◉ Increased level of security of high-quality generated content to distribution

◉ Highly redundant capability

◉ Flexibility to provide various types of Node pools for optimized media workloads

In this particular test, we proved that the intra-Azure network is extremely capable of shipping high bandwidth, latency-sensitive 4K packets from a containerized encoder instance running in West Europe to both East US and Honk Kong Datacenter Regions. This allows the customer to place origin closer to them for further content conditioning.

Workflow:

1. Azure Pipeline is triggered to deploy onto the AKS cluster. In the YAML file (which you can find on Github) there is a reference to the Media Excel Container in Azure Container Registry.

2. AKS starts deployment and pulls container from Azure Container Registry.

3. During Container start custom PHP script is loaded and container is added to the HMS (Hero Management Service). And placed into the correct device pool and job.

4. Encoder loads source and (in this case) push 4K Livestream into Azure Media Services.

5. Media Services packaged Livestream into multiple formats and apply DRM (digital rights management).

6. Azure Content Deliver Network scales livestream.

Scale through Azure Container Instances

With Azure Kubernetes Services you get the power of Azure Container Instances out of the box. Azure Container Instances are a way to instantly scale to pre-provisioned compute power at your disposal. When deploying Media Excel encoding instances to AKS you can specify where these instances will be created. This offers you the flexibility to work with variables like increased density on cheaper nodes for low-cost low priority encoding jobs or more expensive nodes for high throughput high priority jobs. With Azure Container Instances you can instantly move workloads to standby compute power without provisioning time. You only pay for the compute time offering full flexibility for customer demand and future change in platform needs. With Media Excel’s flexible Live/File based encoding roles you can easily move workloads across different compute power offered by AKS and Azure Container Instances.

Azure DevOps pipeline to bring it all together

All the general benefits that come with containerized workload apply in the following case. For this particular proof-of-concept, we created an automated deployment pipeline in Azure DevOps for easy testing and deployment. With a deployment YAML and Pipeline YAML we can easily automate deployment, provisioning and scaling of a Media Excel encoding container. Once DevOps pushes the deployment job onto AKS a container image is pulled from Azure Container Registry. Although container images can be bulky utilizing node side caching of layers any additional container pull is greatly improved down to seconds. With the help of Media Excel, we created a YAML file container pre- and post-container lifecycle logic that will add and remove a container from Media Excel's management portal. This offers an easy single pane of glass management of multiple instances across multiple node types, clusters, and regions.

This deployment pipeline offers full flexibility to provision certain multi-tenant customers or job priority on specific node types. This unlocks the possibility of provision encoding jobs on GPU enabled nodes for maximum throughput or using cheaper generic nodes for low priority jobs.

Azure Media Services and Azure Content Delivery Network

Finally, we push the 4K stream into Azure Media Services. Azure Media Services is a cloud-based platform that enables you to build solutions that achieve broadcast-quality video streaming, enhance accessibility and distribution, analyze content, and much more. Whether you're an app developer, a call center, a government agency, or an entertainment company, Media Services helps you create apps that deliver media experiences of outstanding quality to large audiences on today’s most popular mobile devices and browsers.

Azure Media Services is seamlessly integrated with Azure Content Delivery Network. With Azure Content Delivery Network we offer a true multi CDN with choices of Azure Content Delivery Network from Microsoft, Azure Content Delivery Network from Verizon, and Azure Content Delivery Network from Akamai. All of this through a single Azure Content Delivery Network API for easy provisioning and management. As an added benefit, all CDN traffic between Azure Media Services Origin and CDN edge is free of charge.

With this setup, we’ve demonstrated that Cloud encoding is ready to handle real-time 4K encoding across multiple clusters. Thanks to Azure services like AKS, Container Registry, Azure DevOps, Media Services, and Azure Content Delivery Network, we demonstrated how easy it is to create an architecture that is capable of meeting high throughput time-sensitive constraints.

Continue reading

Six things to consider when using Video Indexer at scale

Your large archive of videos to index is ever-expanding, thus you have been evaluating Microsoft Video Indexer and decided that you want to take your relationship with it to the next level by scaling up.

In general, scaling shouldn’t be difficult, but when you first face such process you might not be sure what is the best way to do it. Questions like “are there any technological constraints I need to take into account?”, “Is there a smart and efficient way of doing it?”, and “can I prevent spending excess money in the process?” can cross your mind. So, here are six best practices of how to use Video Indexer at scale.

1. When uploading videos, prefer URL over sending the file as a byte array

Video Indexer does give you the choice to upload videos from URL or directly by sending the file as a byte array, but remember that the latter comes with some constraints.

First, it has file size limitations. The size of the byte array file is limited to 2 GB compared to the 30 GB upload size limitation while using URL.

Second and more importantly for your scaling, sending files using multi-part means high dependency on your network, service reliability, connectivity, upload speed, and lost packets somewhere in the world wide web, are just some of the issues that can affect your performance and hence your ability to scale.

When you upload videos using URL you just need to give us a path to the location of a media file and we will take care of the rest.

To upload videos using URL via API you can check this short-code sample or you can use AzCopy for a fast and reliable way to get your content to a storage account from which you can submit it to Video Indexer using SAS URL.

2. Increase media reserved units if needed

Usually in the proof of concept stage when you just start using Video Indexer, you don’t need a lot of computing power. Now, when you want to scale up your usage of Video Indexer you have a larger archive of videos you want to index and you want the process to be at a pace that fits your use case. Therefore, you should think about increasing the number of compute resources you use if the current amount of computing power is just not enough.

In Azure Media Services, when talking about computing power and parallelization we talk about media reserved units (RUs), those are the compute units that determine the parameters for your media processing tasks. The number of RUs affects the number of media tasks that can be processed concurrently in each account and their type determines the speed of processing and one video might require more than one RU if its indexing is complex. When your RUs are busy, new tasks will be held in a queue until another resource is available.

We know you want to operate efficiently and you don’t want to have resources that eventually will stay idle part of the time. For that reason, we offer an auto-scale system that spins RUs down when less processing is needed and spin RUs up when you are in your rush hours (up to fully use all of your RUs). You can easily enable this functionality by turning on the autoscale in the account settings or using Update-Paid-Account-Azure-Media-Services API.

To minimize indexing duration and low throughput we recommend you start with 10 RUs of type S3. Later if you scale up to support more content or higher concurrency, and you need more resources to do so, you can contact us using the support system (on paid accounts only) to ask for more RUs allocation.

3. Respect throttling

Video Indexer is built to deal with indexing at scale, and when you want to get the most out of it you should also be aware of the system’s capabilities and design your integration accordingly. You don’t want to send an upload request for a batch of videos just to discover that some of the movies didn’t upload and you are receiving an HTTP 429 response code (too many requests). It can happen due to the fact that you sent more requests than the limit of movies per minute we support. Don’t worry, in the HTTP response, we add a retry-after header. The header we will specify when you should attempt your next retry. Make sure you respect it before trying your next request.

4. Use callback URL

Have you ever called customer service and their response was “I’m now processing your request, it will take a few minutes. You can leave your phone number and we’ll get back to you when it is done”? The cases when you do leave your number and they call you back the second your request was processed are exactly the same concept as using callback URL.

Thus we recommend that instead of polling the status of your request constantly from the second you sent the upload request, you can just add a callback URL, and wait for us to update you. As soon as there is any status change in your upload request, we will send a POST notification to the URL you sent.

You can add a callback URL as one of the parameters of the upload-video API (see below the description from the API). If you are not sure how to do it, you can check the code samples from our GitHub repo. By the way, for callback URL you can also use Azure Functions, a serverless event-driven platform that can be triggered by HTTP and implement a following flow.

5. Use the right indexing parameters for you

Probably the first thing you need to do when using Video Indexer, and specifically when trying to scale, is to think about how to get the most out of it with the right parameters for your needs. Think about your use case, by defining different parameters you can save yourself money and make the indexing process for your videos faster.

We are giving you the option to customize your usage in Video Indexer by choosing those indexing parameters. Don’t set the preset to streaming it if you don’t plan to watch it, don’t index video insights if you only need audio insights, it is that easy.

6. Index in optimal resolution, not highest resolution

Not too long ago, we were in times when HD videos didn't exist. Now, we have videos of varied qualities from HD to 8K. The question is, what video quality do you need for indexing your videos? The higher the quality of the movie you upload means the higher the file size, and this leads to higher computing power and time needed to upload the video.

Our experiences show that, in many cases, indexing performance has almost no difference between HD (720P) videos and 4K videos. Eventually, you’ll get almost the same insights with the same confidence.

For example, for the face detection feature, a higher resolution can help with the scenario where there are many small but contextually important faces. However, this will come with a quadratic increase in runtime and an increased risk of false positives.

Therefore, we recommend you to verify that you get the right results for your use case and to first test it locally. Upload the same video in 720P and in 4K and compare the insights you get. Remember, No need to use a cannon to kill a fly.

Continue reading

Azure Media Services' new AI-powered innovation

Animated character recognition, multilingual speech transcription and more now available

At Microsoft, our mission is to empower every person and organization on the planet to achieve more. The media industry exemplifies this mission. We live in an age where more content is being created and consumed in more ways and on more devices than ever. At IBC 2019, we’re delighted to share the latest innovations we’ve been working on and how they can help transform your media workflows.

Video Indexer adds support for animation and multilingual content

We made our award winning Azure Media Services Video Indexer generally available at IBC last year, and this year it’s getting even better. Video Indexer automatically extracts insights and metadata such as spoken words, faces, emotions, topics and brands from media files, without you needing to be a machine learning expert. Our latest announcements include previews for two highly requested and differentiated capabilities for animated character recognition and multilingual speech transcription, as well as several additions to existing models available today in Video Indexer.

Animated character recognition

Animated content or cartoons are one of the most popular content types, but standard AI vision models built for human faces do not work well with them, especially if the content has characters without human features. In this new preview solution, Video Indexer joins forces with Microsoft’s Azure Custom Vision service to provide a new set of models that automatically detect and group animated characters and allow customers to then tag and recognize them easily via integrated custom vision models. These models are integrated into a single pipeline, which allows anyone to use the service without any previous machine learning skills. The results are available through the no-code Video Indexer portal or the REST API for easy integration into your own applications.

We built these animated character models in collaboration with select customers who contributed real animated content for training and testing. The value of the new functionality is well articulated by Andy Gutteridge, Senior Director, Studio & Post-Production Technology at Viacom International Media Networks, which was one of the data contributors: “The addition of reliable AI-based animated detection will enable us to discover and catalogue character metadata from our content library quickly and efficiently. Most importantly, it will give our creative teams the power to find the content they want instantly, minimize time spent on media management and allow them to focus on the creative.”

Multilingual identification and transcription

Some media assets like news, current affairs, and interviews contain audio with speakers using different languages. Most existing speech-to-text capabilities require the audio recognition language to be specified in advance, which is an obstacle to transcribing multilingual videos. Our new automatic spoken language identification for multiple content feature leverages machine learning technology to identify the different languages used in a media asset. Once detected, each language segment undergoes an automatic transcription process in the language identified, and all segments are integrated back together into one transcription file consisting of multiple languages.

The resulting transcription is available both as part of Video Indexer JSON output and as closed-caption files. The output transcript is also integrated with Azure Search, allowing you to immediately search across videos for the different language segments. Furthermore, the multi-language transcription is available as part of the Video Indexer portal experience so you can view the transcript and identified language by time, or jump to the specific places in the video for each language and see the multi-language transcription as captions as a video is played. You can also translate the output back-and-forth into 54 different languages via the portal and API.

Additional updated and improved models

We are also adding new and improving existing models within Video Indexer, including:

Extraction of people and locations entities

We’ve extended our current brand detection capabilities to also incorporate well-known names and locations, such as the Eiffel Tower in Paris or Big Ben in London. When these appear in the generated transcript or on-screen via optical character recognition (OCR), a specific insight is created. With this new capability, you can review and search by all people, locations and brands that appeared in the video, along with their timeframes, description, and a link to our Bing search engine for more information.

Editorial shot detection model

This new feature adds a set of “tags” in the metadata attached to an individual shot in the insights JSON to represent its editorial type (such as wide shot, medium shot, close up, extreme close up, two shot, multiple people, outdoor and indoor, etc.). These shot-type characteristics come in handy when editing videos into clips and trailers as well as when searching for a specific style of shots for artistic purposes.

Explore and read more about editorial shot type detection in Video Indexer.

Expanded granularity of IPTC mapping

Our topic inferencing model determines the topic of videos based on transcription, optical character recognition (OCR), and detected celebrities even if the topic is not explicitly stated. We map these inferred topics to four different taxonomies: Wikipedia, Bing, IPTC, and IAB. With this enhancement, we now include level-2 IPTC taxonomy.

Tanking advantage of these enhancements is as easy as re-indexing your current Video Indexer library.

New live streaming functionality

We are also introducing two new live-streaming capabilities in preview to Azure Media Services.

Live transcription supercharges your live events with AI

Using Azure Media Services to stream a live event, you can now get an output stream that includes an automatically generated text track in addition to the video and audio content. This text track is created using AI-based live transcription of the audio of the contribution feed. Custom methods are applied before and after speech-to-text conversion in order to improve the end-user experience. The text track is packaged into IMSC1, TTML, or WebVTT, depending on whether you are delivering in DASH, HLS CMAF, or HLS TS.

Live linear encoding for 24/7 over-the-top (OTT) channels

Using our v3 APIs, you can create, manage, and stream live channels for OTT services and take advantage of all the other features of Azure Media Services like live to video on demand (VOD), packaging, and digital rights management (DRM).

New packaging features

Support for audio description tracks

Broadcast content frequently has an audio track that contains verbal explanations of on-screen action in addition to the normal program audio. This makes programming more accessible for vision-impaired viewers, especially if the content is highly visual. The new audio description feature enables a customer to annotate one of the audio tracks to be the audio description (AD) track, which in turn can be used by players to make the AD track discoverable by viewers.

ID3 metadata insertion

In order to signal the insertion of advertisements or custom metadata events on a client player, broadcasters often make use of timed metadata embedded within the video. In addition to SCTE-35 signaling modes, we now also support ID3v2 or other custom schemas defined by an application developer for use by the client application.

Microsoft Azure partners demonstrate end-to-end solutions

Bitmovin is debuting its Bitmovin Video Encoding and Bitmovin Video Player on Microsoft Azure. Customers can now use these encoding and player solutions on Azure and leverage advanced functionality such as 3-pass encoding, AV1/VVC codec support, multi-language closed captions, and pre-integrated video analytics for QoS, ad, and video tracking.

Evergent is showing its User Lifecycle Management Platform on Azure. As a leading provider of revenue and customer lifecycle management solutions, Evergent leverages Azure AI to enable premium entertainment service providers to improve customer acquisition and retention by generating targeted packages and offers at critical points in the customer lifecycle.

Haivision will showcase its intelligent media routing cloud service, SRT Hub, that helps customers transform end-to-end workflows starting with ingest using Azure Data Box Edge and media workflow transformation using Hublets from Avid, Telestream, Wowza and Cinegy, and Make.tv.

SES has developed a suite of broadcast-grade media services on Azure for its satellite connectivity and managed media services customers. SES will show solutions for fully managed playout services, including master playout, localized playout and ad detection and replacement, and 24x7 high-quality multichannel live encoding on Azure.

SyncWords is making its caption automation technology and user-friendly cloud-based tools available on Azure. These offerings will make it easier for media organizations to add automated closed captioning and foreign language subtitling capabilities to their real-time and offline video processing workflows on Azure.

Global design and technology services company Tata Elxsi has integrated TEPlay, its OTT platform SaaS, with Azure Media Services to deliver OTT content from the cloud. Tata Elxsi has also brought FalconEye, its quality of experience (QoE) monitoring solution that focuses on actionable metrics and analytics, to Microsoft Azure.

Verizon Media is making its streaming platform available in beta on Azure. Verizon Media Platform is an enterprise-grade managed OTT solution including DRM, ad insertion, one-to-one personalized sessions, dynamic content replacement, and video delivery. The integration brings simplified workflows, global support and scale, and access to a range of unique capabilities available on Azure.

Many of our partners will also be presenting in the theater at our booth, so make sure you stop by to catch them!

Continue reading

New ways to train custom language models – effortlessly!

Video Indexer (VI), the AI service for Azure Media Services enables the customization of language models by allowing customers to upload examples of sentences or words belonging to the vocabulary of their specific use case. Since speech recognition can sometimes be tricky, VI enables you to train and adapt the models for your specific domain. Harnessing this capability allows organizations to improve the accuracy of the Video Indexer generated transcriptions in their accounts.

Over the past few months, we have worked on a series of enhancements to make this customization process even more effective and easy to accomplish. Enhancements include automatically capturing any transcript edits done manually or via API as well as allowing customers to add closed caption files to further train their custom language models.

The idea behind these additions is to create a feedback loop where organizations begin with a base out-of-the-box language model and improve its accuracy gradually through manual edits and other resources over a period of time, resulting with a model that is fine-tuned to their needs with minimal effort.

Accounts’ custom language models and all the enhancements this blog shares are private and are not shared between accounts.

In the following sections I will drill down on the different ways that this can be done.

Improving your custom language model using transcript updates

Once a video is indexed in VI, customers can use the Video Indexer portal to introduce manual edits and fixes to the automatic transcription of the video. This can be done by clicking on the Edit button at the top right corner of the Timeline pane of a video to move to edit mode, and then simply update the text, as seen in the image below.

The changes are reflected in the transcript, captured in a text file From transcript edits, and automatically inserted to the language model used to index the video. If you were not already using a customer language model, the updates will be added to a new Account Adaptations language model created in the account.

You can manage the language models in your account and see the From transcript edits files by going to the Language tab in the content model customization page of the VI website.

Once one of the From transcript edits files is opened, you can review the old and new sentences created by the manual updates, and the differences between them as shown below.

All that is left is to do is click on Train to update the language model with the latest changes. From that point on, these changes will be reflected in all future videos indexed using that model. Of course, you do not have to use the portal to train the model, the same can be done via the Video Indexer train language model API. Using the API can open new possibilities such as allowing you to automate a recurring training process to leverage ongoing updates.

There is also an update video transcript API that allows customers to update the entire transcript of a video in their account by uploading a VTT file that includes the updates. As a part of the new enhancements, when a customer uses this API, Video Indexer also adds the transcript that the customers uploaded to the relevant custom model automatically in order to leverage the content as training material. For example, calling update video transcript for a video titled "Godfather" will result with a new transcript file named “Godfather” in the custom language model that was used to index that video.

Improving your custom language model using closed caption files

Another quick and effective way to train your custom language model is to leverage existing closed captions files as training material. This can be done manually, by uploading a new closed caption file to an existing model in the portal, as shown in the image below, or by using the create language model and update language model APIs to upload a VTT, SRT or TTML files (similarly to what was done until now with TXT files.)

Once uploaded, VI cleans up all the metadata in the file and strip it down to the text itself. You can see the before and after results in the following table.

Type	Before	After
VTT	NOTE Confidence: 0.891635 00:00:02.620 --> 00:00:05.080 but you don't like meetings before 10 AM.	but you don’t like meetings before 10 AM.
SRT	2 00:00:02,620 --> 00:00:05,080 but you don't like meetings before 10 AM.	but you don’t like meetings before 10 AM.
TTML	<!-- Confidence: 0.891635 --> <p begin="00:00:02.620" end="00:00:05.080">but you don't like meetings before 10 AM.</p>	but you don’t like meetings before 10 AM.

From that point on, all that is left to do is review the additions to the model and click Train or use the train language model API to update the model.

Continue reading

What's new in Azure Media Services video processing

Developers and media companies trust and rely on Azure Media Services for the ability to encode, protect, index, and deliver videos at scale. This week we are proud to announce several enhancements to Media Services including the general availability of the new Azure Media Services v3 API, as well as updates to Azure Media Player.

Low-latency live streaming, 24-hour transcoding, CMAF, and a shiny new API (v3) ready for production

The Azure Media Services v3 API was announced at the Build conference in May 2018, which provided a simplified development model, enabled a better integration experience with key Azure services like Event Grid and Functions, and much more. The API is now generally available and comes with many new exciting features. You can begin migrating workloads built on the preview API over to production use today.

What’s new?

The new Media Services v3 API is a major milestone in the enhancement of the developer experience for Media Services customers. The new API provides a set of SDKs for .NET, .NET Core, Java, Go, Python, Ruby, and Node.js! In addition, the API includes support for the following key scenarios.

Low-latency live streaming with 24-hour transcoding

LiveEvent, the replacement for the Channel entity in the v2 API, now has several major service enhancements.

We often receive the request to lower the latency when streaming live events. Our new low-latency live streaming mode is now available exclusively on the LiveEvent entity in our v3 API. It supports 8 seconds end-to-end latency when used in combination with Azure Media Player’s new low-latency heuristic profile, or ~10 seconds with native HLS playback on an Apple iOS device. Simply configure your live encoder to use smaller 1-second GOP sizes, and you can quickly reduce your overall latency when delivering content to small or medium sized audiences. Of course, it should be noted that the end-to-end latency can vary depending on local network conditions or by introducing a CDN caching layer. Test your exact configuration as your latency could vary.

Looking forward, we will continue to make improvements to our low-latency solution. Last month we announced that we are joining the open source SRT Alliance to help improve low-latency live streaming to Azure with secure and reliable ingest to the cloud. As part of this announcement we have already begun work to add SRT ingest protocol support to our LiveEvent.

To use the new LowLatency feature, you can set the StreamOptionsFlag to LowLatency on the LiveEvent:

Once the stream is up and running, use the Azure Media Player Demo page, and set the the playback options to use the “Low Latency Heuristics Profile”.

Next, when streaming live video, you have two options for long-duration streaming events. If you need to provide linear (24x7x365) live streams, you should use an on-premises encoder with our “pass-through”, non-transcoding LiveEvent. If you require live encoding in the cloud, in the v2 API you were limited to 8 hours of running time. We are very pleased to announce that we have increased support for live transcoding durations up to a full 24 hours when using the new LiveEvent.

Lastly, we have verified several updated RTMP(s)-based encoders including the latest releases from MediaExcel, Telestream Wirecast, Haivision KB, and Switcher Studio.

Easier development with Event Grid and Azure Resource Manager

To make your development experience easier across Azure solutions for media, we are offering more notifications for common operations through Azure Event Grid. You can now subscribe to state change events from Job and JobOutput operations in order to better integrate your custom media applications. If you are creating custom workflows in a Transform, you can specify your own correlation data in the Job object. This correlation data can be extracted from the notifications received through Event Grid to help create workflows that solve common problems with multi-tenant applications, or integration with 3rd-party media asset management systems. When monitoring a live stream, you can use new events such as live ingest heartbeat, connected and disconnected events from the upstream live encoder.

Subscribe to any Media Services event through code, Logic Apps, Functions, or via the Azure portal.

With the transition over to Azure Resource Management (ARM) for our v3 API, you get the following benefits when managing transforms, live events, DRM keys, streaming endpoints, and assets:

1. Easier deployment using ARM templates.

2. Ability to apply role-based access control (RBAC).

Simplified ingest and asset creation

Ingesting content into Media Services used to involve multiple steps such as copying files to Azure Storage, and creating Assets and AssetFiles. In the new API, you can simply point to an existing file in Azure Storage using a SAS URL, or you can ingest from any HTTP(s) accessible URL.

var input = new JobInputHttp(

                     baseUri: "https://nimbuscdn-nimbuspm.streaming.mediaservices.windows.net/2b533311-b215-4409-80af-529c3e853622/",

                     files: new List<String> {"Ignite-short.mp4"}

                     );

We have also simplified the creation of assets in Azure Blob Storage by allowing you to set the container name directly. You can then use the storage APIs to add files into the container. Existing v2 assets will continue to work in the new API, but v3 assets are not backwards compatible.

Streaming and Dynamic Packaging with MPEG CMAF

In the service, we have now released official support for the latest MPEG Common Media Application Format (CMAF) with ‘cbcs’ encryption. CMAF, officially known as MPEG-A Part 19 or ISO/IEC 23000-19, is a new multimedia file format that provides storing and delivery of streaming media using a single encrypted, adaptive bitrate format to a wide range of devices including Apple iPhone, Android, and Windows. Streaming service providers will benefit from this common format through improved interoperability, low-latency streaming, and increased CDN cache efficiency.

To use the new CMAF format, simply add the following new “format=” tag to the URL of your streaming URLs and choose the appropriate manifest type of HLS (for iOS devices) or DASH (for Windows or Android devices).

For MPEG DASH manifest with CMAF format content, use “format=mpd-time-cmaf” as shown below:

https://<<your-account-name>>.streaming.media.azure.net/<<locator-ID>>/<<manifest-name>>.ism/manifest(format=mpd-time-cmaf)

For HLS manifest with CMAF format content use “format=m3u8-cmaf” as shown below:

https://<<your-account-name>>.streaming.media.azure.net/<<locator-ID>>/<<manifest-name>>.ism/manifest(format=m3u8-cmaf)

Manage Media Services through the Command Line

Finally, we have updated the Azure CLI 2.0 module for Media Services to include all features of the v3 API. We will be releasing the final Media Services CLI module on October 23, 2018 for download, or for use directly within the Cloud Shell. The CLI is designed to make scripting Media Services easy. Use the CLI to query for running Jobs, creating Live Events, creating custom Transforms, managing content keys, and more. The CLI module also includes support for Streaming Endpoints, content key policies, and dynamic manifest filters.

Continue reading

Logic Apps, Flow connectors will make Automating Video Indexer simpler than ever

Video Indexer recently released a new and improved Video Indexer V2 API. This RESTful API supports both server-to-server and client-to-server communication and enables Video Indexer users to integrate video and audio insights easily into their application logic, unlocking new experiences and monetization opportunities.

To make the integration even easier, we also added new Logic Apps and Flow connectors that are compatible with the new API. Using the new connectors, you can now set up custom workflows to effectively index and extract insights from a large amount of video and audio files, without writing a single line of code! Furthermore, using the connectors for your integration gives you better visibility on the health of your flow and an easy way to debug it.

To help you get started quickly with the new connectors, we’ve added Microsoft Flow templates that use the new connectors to automate extraction of insights from videos. In this blog, we will walk you through those example templates.

Upload and index your video automatically

This scenario is comprised of two different flows that work together. The first flow is triggered when a new file is added to a designated folder in a OneDrive account. It uploads the new file to Video Indexer with a callback URL to send a notification once the indexing operation completes. The second flow is triggered based on the callback URL and saves the extracted insights back to a JSON file in OneDrive. The reason that two flows are used is to support async upload and indexing of larger files effectively.

Setting up the file upload flow

Navigate to the first template page.

To set up this flow, you will need to provide your Video Indexer API Key and OneDrive credentials.

Once both keys are provided, green marks will appear near your accounts, and you can click to continue to the flow itself and configure it for your needs:

Select a folder that you will place videos in:

Fill in your account Location and ID to get the Video Indexer account token and call the file upload request.

For file upload, you can decide to use the default values, or click on the connector to add additional settings. Notice that you will leave the callback URL empty for now … you’ll add it only after finishing the second template, where the callback URL is created).

Click “Save flow,” and let’s move on to configure the second flow, to extract the insights once the upload completed.

Setting up the JSON extraction flow

Navigate to the second template page.

To set up this flow, you will need to provide your Video Indexer API Key and OneDrive credentials. You will need to update the same parameters as you did for the first flow.

Then, continue to configure the flow.

Fill in your account Location and ID to get your Video Indexer Account Token and the Indexing result.

And select the OneDrive folder to save the insights to. You can also edit the default parameter, if you would like to change the name of the JSON file containing the insights.

Click “Save flow.”

Once the flow is saved, a URL is created in the trigger. Copy the URL from the trigger:

Now, go back to the first flow created and paste the URL in the “Upload video and index” operation under the Callback URL parameter:

Make sure both templates are saved, and you’re good to go!

Try out your newly created flow by adding a video to your OneDrive folder, and go back a few minutes later to see that the insights appear in the destination folder.

Endless integration possibilities

And this is just one example! You can use the new connector for any API call provided by Video Indexer. to upload and retrieve insights, translate the results, get delightful embeddable widgets and even customize your models. Additionally, you can choose to trigger those actions based on different sources like updates to file repositories or emails sent, and to have the results update to our relevant infrastructure or application, or even to generate any number of action items. And you can do all that for large number of files and without coding or having to do a repetitive manual work. Go ahead and try it now for any flow that works for your business needs, it's easy.

Continue reading

Spoken Language Identification in Video Indexer

We are excited to share that Video Indexer has a new capability, Spoken Language Identification (LID)!

A common ask from our customers has been to enable indexing of videos or batches of videos, without manually providing their language. This is especially important for batch uploads. To support this, we have introduced automatic spoken language identification to Video Indexer. The identified language is used to invoke the appropriate speech-to-text model.

LID is based on state of the art Deep Learning applied on the audio. LID currently supports eight languages including English, Chinese, French, German, Italian, Japanese, Spanish, and Russian. It works with high accuracy for high-to-mid quality recordings. We are working on adding more languages to the list, so stay tuned.

Let’s learn more about LID in Video Indexer.

Using LID in Video Indexer

To use the LID capability, you have two options. If you use the portal, you can now select Auto detect in the language selection combo box when uploading a video.

If you use the API to upload a video, use auto as the language parameter value.

Attribute sourceLanguage in the video index JSON under root/videos/insights is assigned the detected language, and attribute sourceLanguageConfidence:

Note to the user

◈ The model behind LID works best with clear recordings: broadcast materials and enterprise materials such as podcasts, lectures, tutorials, etc.

◈ The model may be confused by: noisy recordings, low-quality recordings, highly variant acoustics, and heavy accents.

◈ When the model cannot yield a result with high confidence, VI will fall back to English.

Behind the curtains

Many cognitive tasks, including spoken language identification, are easy for humans but still very challenging for computers. One way to approach this type of tasks, is to mimic the human brain. The initial idea of an artificial neural network was proposed more than 70 years ago. The state of the art in the field is called Deep Learning and is being successfully used for different tasks in Speech and Language Understanding, Computer Vision and even outperforming humans in some tasks, such as diagnosis of skin cancer.

In VI, we harness the power of Deep Learning for Spoken Language Identification. We train the network by presenting it with a huge number of speech examples, coming from different speakers and having diverse acoustic conditions.

The figure below shows how we represent speech to the network. This representation turns voice into an image called a spectrogram. A spectrogram gives us a sense of how complex acoustics are, as 30 seconds of speech can easily require 300,000 pixels!

Legend: Phonetic visualization of a speech smaple. Top: Waveform representation of the recorded audio. Bottom: Spectrogram representation.

The network learns from such huge amounts of data by looking for patterns in the spectrograms, which differentiate between languages. A good pattern is a pattern that is typical to one language. Deep learning starts with a random guess on what these patterns are. Then, using the examples from each language, improves the guess in the right direction.

Continue reading

Azure Media Services announces support for AAD and deprecation of ACS authentication

This month we are announcing the release of support for Azure Active Directory (AAD) authentication in Azure Media Services. Customers of our REST API and .NET client libraries can now use AAD authentication to authorize requests. In addition, we are releasing a new management blade in the Azure Portal to simplify the usage of User and Service Principal authentication with AAD.

With the release of this update to our REST API, we are now able to provide the same role-based access management (RBAC) as provided by the Azure Resource Management (ARM) service. By moving to AAD authentication you will also now be able to track and audit all changes made by specific users or an application connected to your Media Services account. The new Azure Media REST API requires that the user or application making REST API requests must have either contributor or owner level access to the resources it is attempting to manage. More details on how role-based access control works for Azure resources is available at Azure Role-based Access Control.

12-month deprecation notice of ACS authentication support in Azure Media Services

Because Azure Active Directory provides powerful role-based access control features and support for more fine-grained access to resources in your account compared to the ACS token authentication model ("account keys"), we strongly recommend that you update your code and migrate from ACS to AAD-based authentication by June 22, 2018. Also, a key reason for the rapid migration is the upcoming announced deprecation of the ACS key based authentication system.

What does this mean for you?

◉ Microsoft Azure Media Services will end support for Microsoft Azure Access Control Service (ACS)-based authentication on June 22, 2018.
◉ To provide customers sufficient time to update their application code, we are providing 12 months' notice to manage the necessary transition.

What actions should you take?

We recommend that you take the following actions prior to June 22, 2018 to ensure that your applications continue to work as expected:

◉ Update the code for your applications authored for Media Services.
◉ Migrate from ACS-based authentication.
◉ Begin using AAD-based authentication.

Mitigation steps must be taken on or before June 22, 2018 to ensure your applications authored for Media Services using ACS authentication tokens will continue to function as expected without failures in production. Please review each of the new authentication scenarios below closely and take the appropriate action to update to using AAD authentication in your source code.

The Azure Media Services REST API supports authentication for both interactive users and web API, middle-tier, or daemon applications. The following sections provide details on how to use AAD authentication when working directly with the REST API or through the .NET client library.

User Authentication with AAD in Media Services

If you are looking to build a management application for your Azure Media Services account like the Azure Media Services Explorer tool, you can simply login with a User's credentials that has been granted access to the Media Services Resource in the portal via the Access Control (IAM) blade. This type of solution is very useful when you want human interaction with the service that fits one of the following scenarios:

◉ Monitoring dashboard for your Encoding jobs
◉ Monitoring dashboard for your Live Streams
◉ Management application for desktop or mobile users to administer resources in a Media services account.

A native application would first acquire an access token from Azure Active Directory and then use that access token to make all REST API calls. The following diagram shows a typical interactive application authentication flow. For a REST API request to succeed, the calling user must be a “Contributor” or “Owner” of the Azure Media Services account it is trying to access. Unauthorized requests would fail with status code 401. If you see this failure, please double check that you have configured your user as “Contributor” or “Owner” on the Media Services account. You can check this through the Azure portal by searching for your media account and clicking on “Access control” tab.

Users of the .NET client SDK for Media Services must upgrade to the latest version on Nuget (windowsazure.mediaservices version 4.1.0.1 or greater) to use AAD authentication for communicating with REST requests. The following example shows the differences between how to authenticate with the .NET client SDK previously using ACS and the new way that uses AAD credentials.

NOTE: Applications will also need to update their references to include a new assembly "Microsoft.WindowsAzure.MediaServices.Client.Common.Authentication.dll" and add references to that namespace as well as reference to the "Microsoft.IdentityModel.Clients.ActiveDirectory" assembly to get access to the ITokenProvider interface.

DEPRECATED method of authenticating using ACS credentials

// Create and cache Media Services credentials in a static class variable.

_cachedCredentials = new MediaServicesCredentials(

            _mediaServicesAccountName,

            _mediaServicesAccountKey, 

            "urn:windowsazuremediaservices",

            "https://wamsprodglobal001acs.accesscontrol.windows.net");

            

// Used the cached credentials to create CloudMediaContext.

 var mediaContext = new CloudMediaContext(_cachedCredentials);

 mediaContext.Assets.FirstOrDefault();

New method of authenticating using AAD credentials and User authentication

var tokenCredentials = new AzureAdTokenCredentials("{YOUR AAD TENANT DOMAIN HERE}", AzureEnvironments.AzureCloudEnvironment);

var tokenProvider = new AzureAdTokenProvider(tokenCredentials);

var mediaContext = new CloudMediaContext(new Uri("YOUR REST API ENDPOINT HERE"), tokenProvider);

mediaContext.Assets.FirstOrDefault()  // This would return a 401 unauthorized if you are not set up as an authorized user

The "AzureEnvironments.AzureCloudEnvironment" constant is a helper in the .NET SDK to get the right environment variable settings for a public Azure Data Center. It contains pre-defined environment settings for accessing Media Services in the public data centers only. For sovereign or government cloud regions, you can use the " AzureChinaCloudEnvironment", "AzureUsGovernmentEnvrionment", or "AzureGermanCloudEnvironment" respectively.

A lot of the details regarding acquiring an AAD access token has been wrapped and simplified for you in the AzureAdTokenProvider and AzureAdTokenCredentials classes. For example, you do not need to provide the AAD authority, Media services Resource URI or native AAD application details. These are well known values that are already configured by the AAD access token provider class. If you are not using our .NET client SDK, it is recommended to use the ADAL Library to simplify the creation of the access token request using these parameters. The following values are used by default in the AzureAdTokenProvider and AzureAdTokenCredentials classes.

You also have the option of replacing the default implementation of the AzureAdTokenProvider with your own implementation.

AAD Service Principal Authentication in Media Services

For non-human interaction through daemon services, Web APIs, Consumer (mobile or desktop), and Web application, where interactive login or direct user management/monitoring of resources in the Media Services account is not required, you will need to first create an Azure Active Directory application in its own tenant.

Once it is created, you will have to give this application “Contributor” or “Owner” level access to the Media Services account in the Access Control (IAM) blade. Both steps can easily be done through the Azure Portal or through the Azure CLI, or PowerShell script. Note that for AAD resources, “Contributor” has the same access to the resource as “Owner” but only the “Owner” role can grant access to other users. Currently this version of the Media Services REST API does not provide RBAC at the entity level, but that is something we have on the roadmap for our future API update in the Fall. We have also provided the new "API Access" blade in your Media Services account to make it easy to generate the required application or select from an existing one.  If you would like to use x509 certificates instead or ClientID and ClientKey, you can reference the documentation for details on how to configure the SDK

The following examples show how a daemon application may use AAD web application credentials to authenticate requests with the REST service.

Deprecated way of authenticating using ACS credentials

// Create and cache Media Services credentials in a static class variable.

_cachedCredentials = new MediaServicesCredentials(

            _mediaServicesAccountName,

            _mediaServicesAccountKey, 

            "urn:windowsazuremediaservices",

            "https://wamsprodglobal001acs.accesscontrol.windows.net");

            

// Used the cached credentials to create CloudMediaContext.

var mediaContext = new CloudMediaContext(_cachedCredentials);

New way of authenticating with an AAD Service Principal and client symmetric key

var tokenCredentials = new AzureAdTokenCredentials(“{YOUR AAD TENANT DOMAIN HERE}”, new AzureAdClientSymmetricKey(“{YOUR CLIENT ID HERE}”, “{YOUR CLIENT SECRET}”), AzureEnvironments.AzureCloudEnvironment);

var tokenProvider = new AzureAdTokenProvider(tokenCredentials);

var mediaContext = new CloudMediaContext(_mediaServicesApiServerUri, tokenProvider);

mediaContext.Assets.FirstOrDefault();

Making it easy to get started with the new API Access Blade for Media Services

Azure Active Directory authentication could be complex for users unfamiliar with the details of AAD, so we wanted to make it very easy to get started with very little knowledge of AAD. For that reason, we are introducing a new "API Access" blade for Media Services accounts in the portal that will replace the previous ACS "Account keys" blade. We are also disabling the ability to rotate the ACS keys to promote users to update their code and move to AAD support.

The new API Access blade makes the process of connecting to Azure Media Services with AAD much simpler. When you first select the API Access blade, you will be presented with a choice of using either user authentication for human interactive management applications or creating a Service Principal and AAD application for non-human interaction with the Media Services API.

When selecting the user based authentication option, you will see a new panel that contains all the Active Directory information needed to authenticate with the API. This includes the API endpoint that you need to call, along with the ClientID, Domain, and Resource.

For Service Principal authentication, you will see additional values and the ability to select from an existing AAD Application or create a new one directly in the panel.

When the Service Principal blade opens, it selects the first AAD application that meets the following criteria:

◈ It is a registered AAD application

◈ It has "Contributor" or "Owner" RBAC permissions on the account

After creating or selecting an AAD app, you will be able to create and copy a Key (Client Secret) and copy the Client ID (Application ID) which are required to get the access token in this scenario.

In the blade, you can choose to “Create New” AAD Application, or select from an Existing one in your Subscription.  When selecting an existing one, you will see a new blade listing your existing applications to choose from.

Once you select from an existing application or create a new one, you will see additional buttons to “Manage Permissions” or “Manage Application”.  You can use these settings to open the AAD application management blade directly to perform management tasks such as changing keys or reply URL or customizing the applications manifest.

Clicking on the Manage Application button will bring up the AAD application management blade which allows you to create Keys for use with the API using this application.

If you do not have permissions to create AAD apps in your Domain, the AAD app controls of the blade are not shown and a warning message is shown instead.

Next Steps and Actions for Media Services Customers

We are very excited to be making the transition from the older ACS key-based authentication to the more secure, flexible, and role-based Azure Active Directory service. All Azure Media Services customers should begin immediately to migrate to use the new AAD based authentication model by downloading the latest .NET SDK or updating their existing REST-based API calls.

In addition, we are working on a new version of our REST APIs with support for more client SDK languages with AAD authentication. More details on that updated API will come in a later blog post.

The key actions you should be taking today:

1. If you are using.NET, update to the latest SDK and migrate to AAD authentication.

2. Plan early for the deprecation of ACS authentication support in Media Services API. The older ACS authentication support will be shutting off officially on June 22, 2018.

Java SDK and Open Source and Community-driven client SDKs

If you are currently using the Java SDK or one of the many community or open source generated client SDKs for Media Services, you have a couple of options at this time. 

Azure Media Services client SDKs for both JAVA and PHP now support Azure Active Directory (AAD) Authentication. To get the latest Java SDK release see the details in our Java documentation. To download the latest PHP SDK for Media Services, look for version 0.5.7 of the Microsoft/WindowAzure package in the Packagist repository. 

For other open source libraries, since these are not supported directly by the Media Services team, you would need to work with the community SDK developer to prioritize updating the SDK to support AAD for your scenario.

In addition, we are working hard on an updated REST API (v3) that is coming out in 2018 with support for AutoRest generated client SDKs across PHP, Java, Python, and more which will support AAD authentication. We will be following up with more blog posts on migrating to the new v3 API and client SDKs when they are ready for preview.

Continue reading