Conversational Bots Deep Dive – What’s new with the General Availability of Azure Bot Service and Language Understanding

Microsoft brings the latest advanced chatbot capabilities to developers' fingertips, allowing them to create apps that see, hear, speak, understand, and interpret users’ needs -- using natural communication styles and methods.

Today, we’re excited to announce we’re making generally available Microsoft Cognitive Services Language Understanding service (LUIS) and Azure Bot Service, two top notch AI services to create digital agents that interact in natural ways and make sense of the surrounding environment.

Think about the possibilities: all developers regardless of expertise in data science able to build conversational AI that can enrich and expand the reach of applications to audiences across a myriad of conversational channels. The app will be able to understand natural language, reason about content and take intelligent actions. Bringing intelligent agents to developers and organizations that do not have expertise in data science is disruptive to the way humans interact with computers in their daily life and the way enterprises run their businesses with their customers and employees.

Through our preview journey in the past two years, we have learned a lot from interacting with thousands of customers undergoing digital transformation. We highlighted some of our customer stories (such as UPS, Equadex, and more) in our general availability announcement. This post covers conversational AI in a nutshell using Azure Bot Service and LUIS, what we’ve learned so far, and dive into the new capabilities. We will also show how easy it is to get started in building a conversational bot with natural language.

Conversational AI with Azure Bot Service and LUIS

Azure Bot Service provides a scalable, integrated bot development and hosting environment for conversational bots that can reach customers across multiple channels on any device. Bots provide the conversational interface that accepts user input in different modalities including text, speech, cards, or images. The Azure Bot Service offers a set of fourteen channels to interact with users including Cortana, Facebook Messenger, Skype, etc. Intelligence is enabled in the Azure Bot Service through the cloud AI services forming the bot brain that understands and reasons about the user input. Based on understanding the input, the bot can help the user complete some tasks, answer questions, or even chit chat through action handlers. The following diagram summarizes how conversational AI applications are enabled through the Azure Bot Service and the Cloud AI services including language understanding, speech recognition, Q&A Maker, etc.

Language Understanding (LUIS) is the key part of the bot brain that allows the bot to understand natural language input and reason about it to take the appropriate action. As customization is critical for every business scenario, Language Understanding helps build custom models for your business vertical with little effort and without prior expertise in data science. Designed to identify valuable information in conversations, it interprets user goals (intents) and distills valuable information from sentences (entities), for a high quality, nuanced language model.

With the General Availability of Language Understanding and Azure Bot Service, we're also introducing new capabilities to help you achieve more and delight your users

Language Understanding:

• With an updated user interface, we’re providing Language Understanding service (LUIS) users more intents and entities than ever: expanding up to 500 intents (task or action identified in the sentence) and 100 entities (relevant information extracted, from the sentence, to complete the task or action associated to the intent) per application.
• Language Understanding is now available in 7 new regions (South Central US, East US, West US 2, East Asia, North Europe, Brazil South, Australia East) on top of the 5 existing regions (West Europe, West US, East US2, West central US, South east Asia). This will help customers to improve network latency and bandwidth.
• The Language Understanding service is also supporting more languages for its various features, in addition to English.
• The prebuilt entities (representing common concepts like numbers, date, time) previously available in English are now available in French and Spanish.
• Prebuilt domains (off-the-shelf collections of intents and entities grouped by domain that you can directly add and use in your application) are now also available in Chinese.
• Phrase suggestions that help the developer customize your LUIS domain vocabulary are available in 7 new languages Chinese, Spanish, Japanese, French, Portuguese, German, and Italian.

Azure Bot Service:

• Speed bot development by providing an integrated environment with the Microsoft Bot Framework channels, development tools and hosting solutions.
• Connect with your audience with no code modifications via our supported channels on the Bot Service; Office 365 Email, GroupMe, Facebook Messenger, Kik, Skype, Slack, Microsoft Teams, Telegram, text/SMS, Twilio, Cortana, Skype for Business – or provide a custom experience in your app or website.
• Bot Service is now integrated into the Azure portal; easy access to 24x7 support, monitoring capabilities, integrated billing and more in the trusted Azure ecosystem.
• Now generally available in 9 different regions namely West US, East US, West Europe, and Southeast Asia including new deployments in North Europe, Australia Southeast, Australia East, Brazil South, and East Asia regions.
• We are also announcing Premium Channels including webchat and directline.  Premium channels offer unique capabilities over the standard channels:
• Communicate with your users on your website or in your application instead of sharing that data with public chat services.
• Open Source webchat and directline clients enabling advanced customization opportunities.
• 99.9% availability guarantees for premium channels

Developers can connect to other Azure services to enrich their bots as well as add Cognitive Services to enable your bots to see, hear, interpret, and interact in more human ways. For example, on top of language, the Computer Vision and Face APIs can enable bots to understand images and faces passed to the bot.

Learning through our customer’s experiences

For several years now, Microsoft has been leading the charge into the application of AI to build new intelligent conversational experiences…everything from proprietary solutions built to target a specific audience on a specific chat service to general purpose API’s that expect the developer to create the rest of the custom solution themselves.  We are still at the beginning of this evolution of the conversational application model; but already we have takeaways that are guiding how we think about the future.

Bots are changing how we do business. We are constantly having great discussions with customers who see bots as a key part of their digital transformation as a business. They see the opportunity to enhance their customer support experiences, provide easy access to information, or even expose their business to an audience that might not otherwise visit their website.

Developers need to have choice in technologies. With the growth in popularity of open source technologies, developers want choice of the technology components they use to build solutions.

Great conversational applications are multi-modal. Our customers are building conversational experiences which accomplish multiple tasks. For example, a customer support bot may have a Q&A search function, a support ticket entry function, a guided dialog to diagnose a problem, and an appointment scheduling function that hands off to a human for final confirmation.

AI platforms must scale to the needs of business. Often as not, business scenarios are based on sets of concepts that are being codified into the bot. Developers require the technologies they depend on to scale to the complexity of their business without arbitrary limits getting in the way.

Conversational app platforms need to be reliable and compliant. In the same way that mobile app platforms have needed to provide robust and secure platforms to enable great productivity scenarios, so too will conversational application platforms; they must be certifiably secure, reliable, compliant and privacy aware. In addition, the platform should to make it easy for developers building to it to build compliant solutions as well.

Businesses are global and multi-lingual. Businesses need to talk to customers world-wide 24/7 in their language of choice.

There is art in building a great conversational application. Much in the same way the 80’s and 90’s cemented what we now think of as common controls for native apps, and the 2000’s for web and mobile, the industry is still defining what it means to be a great conversational application.

Key design considerations

Given the learnings we’ve had, we’ve anchored our design on the following six points to shape the Azure Bot Service and Language Understanding (LUIS) capabilities:

Code-first approach: Azure Bot Service is built on top of the BotBuilder SDK V3 (in Node.js) that takes a code-first approach to enable developers to have full control over their bots’ conversational capabilities. Available for both Node.JS and C#, the open source SDK’s provides multiple dialog types and conversational orchestration tools to help the developer with various tasks like slot filling, dialog management and card representation.

Different dialog management flavors: developers build bots that range from simple question answer bots to multi-turn solutions that span ten or fifteen turns to complete a task. We provide a rich set of dialog management flavors to cover the different task types a bot developer might wish to expose. You can create bots that utilize a mix of prompts, form filling, natural language, and your own dialog management system with the ability to reuse some of the components like prompts.

Open bot platform: Building on Azure's commitment to open source technologies, applications using our SDK and LUIS can be deployed on any connected infrastructure and consumed from any device anywhere targeting your audience on multiple chat channels. This open design allows the offering to be integrated with different deployment platforms including public cloud or on-premise infrastructure.

Global and multi-lingual: We have put considerable effort into making our services highly available and as close to customers as possible as part of the Azure cloud.  Azure Bot Service and Language Understanding support a growing list of languages for understanding conversations.

Getting started quickly: While bots can be deployed anywhere, with Azure we provide rich connected cloud services for hosting your bot and AI applications with a single click.  The Azure Bot Service and LUIS get you a running bot that can converse with users in a natural way in minutes. Azure Bot Service takes care of provisioning all of the Azure resources you need so that developers can focus on their business logic. LUIS provides customizable pre-built apps and entity dictionaries, such as Calendar, Music, and Devices, so you can build and deploy a solution more quickly. Dictionaries are mined from the collective knowledge of the web and supply billions of entries, helping your model to correctly identify valuable information from user conversations.

Custom models with little effort: as customization is critical for every business scenario, LUIS capitalizes on the philosophy of machine teaching to help non-expert machine learning developers build effective custom language understanding models. While machine learning focuses on creating new algorithms and improving the accuracy of “learners”, the machine teaching discipline focuses on the efficacy of the “teachers”. Machine teaching as a discipline is a paradigm shift that follows and extends principles of software engineering and programming languages. It provides the developer with a set of tools to build machine learning models by transferring the developer domain knowledge to the machine learning algorithms. This contrasts with Machine Learning which is about creating useful models from this knowledge. Developer knowledge is expressed in LUIS through schema (what intents and entities are in the LUIS application) and labeled examples.  It supports a wide variety of techniques for reliably recognizing entities with normalization to allow them to be easily consumed in a program.

Always monitor, learn and improve: Azure Bot Service and LUIS use Azure monitoring tools to help developers monitor the performance of their bots including the quality of the language understanding models and the bot usage. Once the model starts processing input, LUIS begins active learning, allowing you to constantly update and improve the model. It helps you pick the most informative utterances from your real bot traffic to add to your model and continuously improve. This intelligent selection of examples to add to the training data of the LUIS model helps developers build cost effective models that don’t require a lot of data and yet perform with high accuracy.

Getting started with the Bot Service and Language Understanding

In this section, we’ll create a bot using the Azure Bot Service that uses Language Understanding (LUIS) to understand the user. When creating a bot using natural language, the bot determines what a user wants to do by identifying their intent. This intent is determined from spoken or textual input, or utterances, which in turn can be mapped to actions that Bot developers has coded. For example, a note-taking bot recognizes a Notes. Create intent to invoke the functionality for creating a note. A bot may also need to extract entities, which are important words in utterances. In the example of a note-taking bot, the Notes. Title entity identifies the title of each note.

Create a Language Understanding bot with Bot Service

To create your bot; log in the Azure portal, select Create new resource in the menu blade and select AI + Cognitive Services.

You can browse through the suggestions, or search for Web App Bot.

Once selected, the Bot Service blade should appear; which will be familiar to users of Azure services. For those that aren’t, here you can specify information about your service for the Bot Service to use in creating your bot such as where it will live, what subscription in and so forth. In the Bot Service blade, provide the required information, and click Create. This creates and deploys the bot service and LUIS app to Azure. Some interesting fields:

• Set App name to your bot’s name. The name is used as the subdomain when your bot is deployed to the cloud (for example, mynotesbot.azurewebsites.net). This name is also used as the name of the LUIS app associated with your bot. Copy it to use later, to find the LUIS app associated with the bot.
• Select the subscription, resource group, hosting plan, and location.
• For pricing, you can choose the free pricing tier. You can go back and change that at any time if you need more.
• For this sample, select the Language understanding (C#) template for the Bot template field.
• For the final required field, choose the Azure Storage where you wish to store your bot’s conversation state. Think of this as where the bot keeps track of where each user is in the conversation.

Now that you’re complete, you can click Create. Azure will set about creating your bot including the resources it needs to operate your bot and a LUIS account to host your natural language model. Once complete, you’ll receive a notification via the bell in the top right corner of the Azure portal.

Next up, lets confirm that the bot service has been deployed.

◉ Click Notifications (the bell icon that is located along the top edge of the Azure portal). The notification will change from Deployment started to Deployment succeeded.

◉ After the notification changes to Deployment succeeded, click Go to resource on that notification.

Try the bot

So now you should have a working bot. Let’s try it out.

Once the bot is registered, click Test in Web Chat to open the Web Chat pane. Type "hello" in Web Chat.

The bot responds by saying "You have reached Greeting. You said: hello". This confirms that the bot has received your message and passed it to a default LUIS app that it created. This default LUIS app detected a Greeting intent.

Note: Occasionally, the first message or two after startup may need to be retried before the bot will answer.

Viola! You have a working bot! The default bot only knows a few things; it recognizes some greetings, as well as help and cancel. In the next section we’ll modify the LUIS app for our bot to add some new intents for our Note taking bot.

Modify the LUIS app

Log in to www.luis.ai using the same account you use to log in to Azure. Click on My apps. If all has gone well, in the list of apps, you’ll find the app with the same name as the App name from the Bot Service blade when you created the Bot Service.

After opening the app, you should see it has four intents: Cancel, Greeting, Help, and None. The first three we already mentioned. None is a special intent in LUIS that captures “everything else”.

For our sample, we’re going to add three intents for the user: Note.Create and Note.ReadAloud. Conveniently, one of the great features about LUIS are the pre-built domains that can be used to bootstrap your application, of which Note is one.

• Click on Pre-built Domains in the lower left of the page. Find the Note domain and click Add domain.
• This tutorial doesn't use all the intents included in the Note prebuilt domain. In the Intents page, click on each of the following intent names and then click the Delete Intent button to remove them from your app.
• Note.ShowNext
• Note.DeleteNoteItem
• Note.Confirm
• Note.Clear
• Note.CheckOffItem
• Note.AddToNote
• Note.Delete

◉ IMPORTANT: The only intents that should remain in the LUIS app are the Note.ReadAloud, Note.Create, None, Help, Greeting, and Cancel intents.  If they’re still there, your app will still work, but may more often behave inconsistently.

As mentioned earlier, the Intents that we’ve now added represent the types of things we expect the user to want the bot to do.  Since these are pre-defined, we don’t have to do any further tuning to the model, so let’s jump right to training and publishing your model.

◈ Click the Train button in the upper right to train your app.  Training takes everything you’ve entered into the model by creating intents and entities, entering utterances and labeling them and generates a machine learned model, all with one click.  You can test your app here in the LUIS portal, or move on to publishing so that it’s available to your bot.

◈ Click PUBLISH in the top navigation bar to open the Publish page. Click the Publish to production slot button. After successful publish, copy the URL displayed in the Endpoint column the Publish App page, in the row that starts with the Resource Name Starter_Key. Save this URL to use later in your bot’s code. The URL has a format similar to this example: https://westus.api.cognitive.microsoft.com/luis/v2.0/apps/xxxxxxxxxxxxxxxxx?subscription-key=xxxxxxxxxxxxxx3&timezoneOffset=0&verbose=true&q=

Your Language Understanding Application is now ready for your Bot. If the user asks to create, delete, or read back a note, Language Understanding will identify that and return the correct intent to the Bot to be acted on. In the next section we’ll add logic to the bot to handle these Intents.

Modify the bot code

The Bot Service is set up to work in a traditional development environment; sync your source code with GIT and work in your favorite dev environment. That said, Azure Bot Service also offers the ability to edit right in the portal; which is great for our experiment. Click Build and then click Open online code editor.

First, some preamble. In the code editor, open BasicLuisDialog.cs. It contains the code for handling Cancel, Greeting, Help, and None intents from the LUIS app.

Add the following statement:

using System.Collections.Generic;

Create a class for storing notes

Add the following after the BasicLuisDialog constructor:

private readonly Dictionary<string, Note> noteByTitle = new Dictionary<string, Note>();

private Note noteToCreate;

private string currentTitle;

// CONSTANTS

// Name of note title entity

public const string Entity_Note_Title = "Note.Title";

// Default note title

public const string DefaultNoteTitle = "default";

[Serializable]

public sealed class Note : IEquatable<Note>

{

public string Title { get; set; }

public string Text { get; set; }

public override string ToString()

{

return $"[{this.Title} : {this.Text}]";

}

public bool Equals(Note other)

{

return other != null

&& this.Text == other.Text

&& this.Title == other.Title;

}

public override bool Equals(object other)

{

return Equals(other as Note);

}

public override int GetHashCode()

{

return this.Title.GetHashCode();

}

}

Handle the Note.Create intent

Note.Create intent, add the following code to the BasicLuisDialog class.

[LuisIntent("Note.Create")]

public Task NoteCreateIntent(IDialogContext context, LuisResult result)

{

EntityRecommendation title;

if (!result.TryFindEntity(Entity_Note_Title, out title))

{

// Prompt the user for a note title

PromptDialog.Text(context, After_TitlePrompt, "What is the title of the note you want to create?");

}

else

{

var note = new Note() { Title = title.Entity };

noteToCreate = this.noteByTitle[note.Title] = note;

// Prompt the user for what they want to say in the note

PromptDialog.Text(context, After_TextPrompt, "What do you want to say in your note?");

}

return Task.CompletedTask;

}

private async Task After_TitlePrompt(IDialogContext context, IAwaitable<string> result)

{

EntityRecommendation title;

// Set the title (used for creation, deletion, and reading)

currentTitle = await result;

if (currentTitle != null)

{

title = new EntityRecommendation(type: Entity_Note_Title) { Entity = currentTitle };

}

else

{

// Use the default note title

title = new EntityRecommendation(type: Entity_Note_Title) { Entity = DefaultNoteTitle };

}

// Create a new note object

var note = new Note() { Title = title.Entity };

// Add the new note to the list of notes and also save it in order to add text to it later

noteToCreate = this.noteByTitle[note.Title] = note;

// Prompt the user for what they want to say in the note

PromptDialog.Text(context, After_TextPrompt, "What do you want to say in your note?");

}

private async Task After_TextPrompt(IDialogContext context, IAwaitable<string> result)

{

// Set the text of the note

noteToCreate.Text = await result;

await context.PostAsync($"Created note **{this.noteToCreate.Title}** that says \"{this.noteToCreate.Text}\".");

context.Wait(MessageReceived);

}

Handle the Note.ReadAloud Intent

The bot can use the Note.ReadAloud intent to show the contents of a note, or of all the notes if the note title isn't detected. Paste the following code into the BasicLuisDialog class.

[LuisIntent("Note.ReadAloud")]

public async Task NoteReadAloudIntent(IDialogContext context, LuisResult result)

{

Note note;

if (TryFindNote(result, out note))

{

await context.PostAsync($"**{note.Title}**: {note.Text}.");

}

else

{

// Print out all the notes if no specific note name was detected

string NoteList = "Here's the list of all notes: \n\n";

foreach (KeyValuePair<string, Note> entry in noteByTitle)

{

Note noteInList = entry.Value;

NoteList += $"**{noteInList.Title}**: {noteInList.Text}.\n\n";

}

await context.PostAsync(NoteList);

}

context.Wait(MessageReceived);

}

public bool TryFindNote(string noteTitle, out Note note)

{

// TryGetValue returns false if no match is found.

bool foundNote = this.noteByTitle.TryGetValue(noteTitle, out note);

return foundNote;

}

public bool TryFindNote(LuisResult result, out Note note)

{

note = null;

string titleToFind;

EntityRecommendation title;

if (result.TryFindEntity(Entity_Note_Title, out title))

{

titleToFind = title.Entity;

}

else

{

titleToFind = DefaultNoteTitle;

}

// TryGetValue returns false if no match is found.

return this.noteByTitle.TryGetValue(titleToFind, out note);

}

Build the bot

Now that the cut and paste part is done, you can right-click on build.cmd in the code editor and choose Run from Console. Your bot will be built and deployed from within the online code editor environment.

Test the bot

In the Azure Portal, click on Test in Web Chat to test the bot. Try type messages like "Create a note", "read my notes", and "delete notes".  Because you’re using natural language you have more flexibility on how you state your request, and in turn, Language Understanding’s Active Learning feature can be used such that you can open your Language Understanding application and it can make suggestions about things you said which it didn’t understand and might make your app more effective.

Tip: If you find that your bot doesn't always recognize the correct intent or entities, improve your Language Understanding app's performance by giving it more example utterances to train it. You can retrain your Language Understanding app without any modification to your bot's code.