← Back to Tutorials Chapter 11

Custom Question Answering

Evolution from QnA Maker to Custom Question Answering

Custom Question Answering is the Azure AI service that creates a conversational layer over structured and unstructured content. You give it a set of question-and-answer pairs, FAQ pages, product manuals, or support documents, and it returns the most relevant answer to a user's natural language question. This service was originally called QnA Maker and lived at qnamaker.ai. Microsoft has since merged it into Azure AI Language Studio under the name Custom Question Answering, aligning it with the other custom language features covered in the previous chapters.

The migration from QnA Maker to Custom Question Answering brings several improvements. The underlying ranking model now uses transformer-based architecture for better semantic matching. The management portal is unified within Language Studio alongside CLU and custom text projects. The API surface is consolidated into the Azure AI Language SDK rather than a separate QnA Maker SDK. The bot integration pattern is also updated — new bots should use the Language SDK's conversational question answering client instead of the older QnA Maker connector. QnA Maker knowledge bases can be migrated to Custom Question Answering using the built-in export-and-import flow in Language Studio.

Feature	QnA Maker (Legacy)	Custom Question Answering
Portal	qnamaker.ai	Language Studio
SDK	Microsoft.Azure.CognitiveServices.Knowledge.QnAMaker	Azure.AI.Language.QuestionAnswering
Ranking model	Older neural network	Transformer-based semantic ranking
Multi-turn	Limited	Enhanced with context flow
Active learning	Separate process	Integrated in Language Studio
Synonyms	Manual synonym list	Alternate phrases per QnA pair
Chit-chat	Built-in personality sets	Same, but managed in Language Studio

When to Use Question Answering vs. CLU vs. AI Search

A common point of confusion on the AI-102 exam is knowing which service to use for a given scenario. Question Answering, Conversational Language Understanding (CLU), and Azure AI Search all retrieve information from a knowledge source, but they are designed for fundamentally different interactions. Question Answering returns the best-matching answer string from a predefined knowledge base. It is ideal for FAQ-style interactions, help desk support, and internal knowledge bases where the set of possible answers is known in advance. The user asks a question in natural language and the service returns the single most relevant answer.

CLU predicts an intent and extracts entities from an utterance. It does not retrieve information from a knowledge base. Instead, it classifies the user's intent so your application can respond appropriately. A CLU model might tell you that the user wants to "CheckStoreHours," but it does not know what those hours are. Your application code must supply the actual answer. AI Search returns ranked document results from an indexed set of documents. It is the right choice when the user needs to search across unstructured or semi-structured content — legal documents, product catalogs, research papers — and the answer might span multiple documents or require reading the full content.

Criterion	Question Answering	CLU	AI Search
Returns	A single answer string	Intent + entities	Ranked document list
Knowledge source	QnA pairs, FAQs, documents	Example utterances only	Indexed document corpus
Best for	FAQ bots, help desks	Command-like interactions	Full-text search over documents
Conversation	Multi-turn follow-ups	Single-turn intent mapping	No built-in conversation
Example	"What is your return policy?"	"Book a flight to Seattle"	"Show me all contracts signed in 2024"

Creating a Question Answering Project in Language Studio

To create a Custom Question Answering project, navigate to language.cognitive.azure.com, select your Language resource, and click "Custom question answering" under the "Answer questions" section. Click "Create new project," provide a name such as "SupportKB," and select the language. You can also choose to enable multiple languages in a single project, which is useful if you handle FAQs in several languages and want to maintain one knowledge base.

The project workspace in Language Studio has several tabs: Manage source (where you add FAQ URLs, files, or unstructured content), Edit knowledge base (where you manually manage QnA pairs), Test (where you interactively test the knowledge base), Deploy (to publish), and the Analytics tab for usage metrics. The interface is designed to guide you from creating content through deploying as a bot endpoint.

Adding Question-and-Answer Pairs Manually

You can add QnA pairs directly in the Edit knowledge base tab. Click "Add QnA pair" and fill in the question, answer, and optional metadata fields. Each pair can have multiple alternate questions that express the same information need in different wording. For example, the question "What is your return policy?" might also be phrased as "How do I return an item?", "Can I get a refund?", and "What is the returns process?" The more alternate questions you add, the better the semantic matcher will be at finding the right answer.

Metadata tags let you filter answers by category. For a support knowledge base, you might tag pairs with metadata like "category: billing," "category: shipping," or "product: widget-a." When querying the API, you can pass a metadata filter so the service only searches within a specific category. This is especially useful when the knowledge base is large and answers need to be scoped to the context of the user's current session.

Importing from FAQ Pages, Product Manuals, and Support Documents

Rather than typing every pair by hand, you can import content from several source types. The most common is a FAQ page URL. Language Studio crawls the URL, extracts question-and-answer patterns (typically <h2> or <h3> headings followed by paragraph content), and automatically creates QnA pairs. You can review and edit the imported pairs after the crawl completes. For structured FAQ pages, this works well with minimal post-processing.

Product manuals and support documents can be uploaded as PDF, DOCX, TXT, or Excel files. Language Studio parses the document and extracts sections that look like question-answer pairs. For unstructured documents where the QnA format is not obvious, the service creates pairs based on the document's heading structure. Each heading becomes a question and the content under it becomes the answer. Review the results carefully — the automatic extraction is a starting point, not a finished knowledge base. You will need to clean up pairs where the heading is not actually a question or where the answer content is too long or too short.

Training and Testing a Knowledge Base

Custom Question Answering does not have an explicit "train" button like CLU does. Instead, the model is updated automatically as you add, edit, or reorder QnA pairs. When you save changes in the Edit knowledge base tab, the service reindexes the content and updates the ranking model. You can then go to the Test tab to interactively query the knowledge base and see the returned answers with confidence scores.

The Test pane shows your question, the top answer, its confidence score, and the source of the answer. You can also see the alternate phrasing and follow-up prompts associated with the matched pair. If the model returns a low-confidence answer (below 30-40%), consider adding more alternate questions for that pair or checking whether the answer content is specific enough to match the question.

Publishing and Creating a Web Chat Bot

When you are satisfied with the knowledge base, click the "Deploy" tab and select the deployment slot (staging or production). Deployment makes the knowledge base available at a REST endpoint. From the same tab, you can create a web chat bot by clicking "Create a bot." This opens the Azure portal and creates an Azure Web App Bot connected to your knowledge base. The bot uses a simple QnA dialog that routes all user messages to the question answering endpoint and returns the answer. This is the fastest way to get a working conversational bot with zero code.

Multi-Turn Conversations: Follow-Up Prompts and Context

Real-world customer support rarely fits a single question-and-answer interaction. A user asks "How do I return an item?" and the answer is "Items can be returned within 30 days." The user then asks "What if I lost my receipt?" A flat knowledge base has no way to know this second question is a follow-up to the first. Multi-turn conversation solves this by letting you define follow-up prompts on specific QnA pairs. Each prompt is a short question that the bot suggests as a possible next query, and the answer for that prompt is scoped to the context of the parent pair.

In the Edit knowledge base tab, select a QnA pair and scroll to the "Follow-up prompts" section. Click "Add follow-up prompt," enter the prompt text (e.g., "What if I lost my receipt?"), and link it to the target QnA pair that contains the answer. When the bot returns the parent answer, it includes the follow-up prompt as a suggested action. If the user clicks the prompt, the bot passes context indicating that this is a follow-up to the original question, and the service uses that context to improve answer matching. The context is passed as a conversationId or previousQnaId parameter in the API call.

Active Learning: Improving from User Interactions

Active learning in Question Answering works differently than in CLU. When you enable it, the service monitors user queries and detects patterns where the returned answer was not the intended one. This is inferred from implicit signals — if a user asks a question, sees the answer, and then immediately asks the same question again, it suggests the first answer was not useful. The active learning suggestions appear in a review interface where you can add new alternate questions to the correct QnA pair.

To use active learning, enable it in the project settings in Language Studio. Integrate the "train" API into your application so that every user interaction sends feedback to the service about whether the returned answer was helpful. The service analyzes the feedback patterns and periodically generates suggestions. Review the suggestions in Language Studio, accept or reject each one, and save the knowledge base to apply the changes.

Alternate Phrasing for Better Matching

The semantic ranker that powers Question Answering uses transformer-based models to understand the meaning behind a question, not just keyword overlap. However, the model still benefits significantly from alternate phrasing. Each alternate question you add gives the model another representation of the same information need, improving its ability to match diverse user inputs. For each primary question, aim for 5-10 alternate phrasings that vary in word choice, sentence structure, and length. Include formal and informal versions, short and long versions, and phrasings that use synonyms for key terms.

Adding Chit-Chat for Personality

A knowledge base that only answers product support questions feels robotic when a user says "Hello" or "Thanks." Chit-chat adds a set of conversational QnA pairs covering greetings, farewells, small talk, and jokes. Language Studio includes built-in chit-chat personality sets: Professional, Friendly, and Witty. Each set contains dozens of prebuilt QnA pairs for common conversational exchanges. When you add chit-chat to your project, these pairs are imported into your knowledge base and compete with your domain pairs. If the user says "Hello," the chit-chat pairs will match with high confidence, and your bot responds naturally. You can also edit the chit-chat pairs or create your own to match your brand voice.

Exporting and Backing Up Knowledge Bases

Knowledge bases created in Language Studio can be exported as a TSV or Excel file. The export contains all QnA pairs, alternate questions, metadata, and follow-up prompt definitions. To export, go to the project settings in Language Studio and click "Export." Choose the format and the file downloads. To restore a knowledge base in a different resource, create a new project and use the "Import" option to upload the export file. Note that trained model weights are not included in the export — the knowledge base content itself determines the model behavior, and the semantic ranker is the same base model regardless of export.

Multi-Language Question Answering Strategies

If your application needs to support multiple languages, you have two approaches. The first is a single multi-language project where all QnA pairs are in the project's primary language, and the service translates incoming queries on the fly. This is simpler to manage but relies on machine translation quality. The second approach is separate projects per language, each with human-created QnA pairs in the target language. This produces higher quality answers but requires maintaining multiple knowledge bases. The multi-language setting is configured at project creation time and cannot be changed later, so plan ahead.

Code Sample: REST API Call to QA Endpoint

The following Python example queries a published knowledge base using the Azure AI Language SDK:

import os
from azure.core.credentials import AzureKeyCredential
from azure.ai.language.questionanswering import QuestionAnsweringClient

endpoint = os.environ["LANGUAGE_ENDPOINT"]
key = os.environ["LANGUAGE_KEY"]
project_name = "SupportKB"
deployment_name = "production"

client = QuestionAnsweringClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(key)
)

question = "How do I return an item?"
response = client.get_answers(
    question=question,
    project_name=project_name,
    deployment_name=deployment_name
)

print(f"Question: {question}")
print(f"Answer: {response.answers[0].answer}")
print(f"Confidence: {response.answers[0].confidence:.2%}")

# Query with metadata filter
response = client.get_answers(
    question=question,
    project_name=project_name,
    deployment_name=deployment_name,
    filters={
        "metadataFilter": {
            "metadata": [
                {"key": "category", "value": "returns"}
            ]
        }
    }
)

# Query with context for multi-turn
response = client.get_answers(
    question="What if I lost my receipt?",
    project_name=project_name,
    deployment_name=deployment_name,
    answer_context={
        "previousQnaId": 42,
        "previousUserQuery": "How do I return an item?"
    }
)

// C# - querying a knowledge base
using Azure;
using Azure.AI.Language.QuestionAnswering;

var endpoint = Environment.GetEnvironmentVariable("LANGUAGE_ENDPOINT");
var key = Environment.GetEnvironmentVariable("LANGUAGE_KEY");
var projectName = "SupportKB";
var deploymentName = "production";

var client = new QuestionAnsweringClient(
    new Uri(endpoint),
    new AzureKeyCredential(key));

var question = "How do I return an item?";
Response<AnswersResult> response = client.GetAnswers(
    question,
    projectName,
    deploymentName);

Console.WriteLine($"Answer: {response.Value.Answers[0].Answer}");
Console.WriteLine($"Confidence: {response.Value.Answers[0].Confidence:P2}");

Summary

Custom Question Answering converts structured and unstructured content into a conversational knowledge base. It is the successor to QnA Maker and is now managed within Azure AI Language Studio. The service supports importing from FAQ URLs and documents, manual QnA pair creation with alternate phrasing, multi-turn conversations with follow-up prompts, active learning from user interactions, and chit-chat for conversational personality. The query API is accessed through the QuestionAnsweringClient in the Azure AI Language SDK and returns the best-matching answer with a confidence score. For the AI-102 exam, understand the differences between Question Answering, CLU, and AI Search, and know the migration path from QnA Maker to Custom Question Answering. The next chapter covers Azure Speech-to-Text for transcribing audio into text.