Natural Language Processing

Azure AI Language Services Overview

Azure AI Language provides prebuilt natural language processing capabilities that analyze text without requiring custom model training. The service exposes REST APIs and client SDKs for a range of text analysis tasks including sentiment analysis, key phrase extraction, named entity recognition, language detection, and personally identifiable information (PII) detection. These capabilities are part of the Language service under the Azure AI Services umbrella.

The Language service can be accessed through a single-service Language resource or through the multi-service Cognitive Services resource. The API endpoint follows the pattern https://<name>.cognitiveservices.azure.com/language/. On the AI-102 exam, these capabilities fall under the "Implement natural language processing" domain, which carries a 20-25% weight. You should know which API to call for each text analysis scenario and how to interpret the JSON responses.

Key Phrase Extraction

Key phrase extraction identifies the main talking points in a document. The service returns a list of phrases that capture the essential topics, which is useful for summarizing large volumes of text, generating tags for content management systems, and identifying trends in customer feedback.

The API accepts a batch of documents — up to 10 per request for the standard tier — and returns key phrases for each document independently. Each document can be up to 5,120 characters. The key phrases are returned as an array of strings, sorted by relevance. Unlike entity recognition, key phrases are not typed or classified; they are simply topical words and phrases extracted from the text.

# Python - key phrase extraction
import os
from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import AzureKeyCredential

endpoint = os.environ["LANGUAGE_ENDPOINT"]
key = os.environ["LANGUAGE_KEY"]

client = TextAnalyticsClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(key)
)

documents = [
    "The new Azure AI services update includes improved "
    "sentiment analysis and faster key phrase extraction. "
    "Customers have reported a 30% reduction in processing time.",
    "Our support team handled over 10,000 tickets last month "
    "with an average resolution time of 4 hours."
]

results = client.extract_key_phrases(documents)

for i, doc in enumerate(results):
    if not doc.is_error:
        print(f"Document {i+1} key phrases:")
        for phrase in doc.key_phrases:
            print(f"  - {phrase}")
    else:
        print(f"Error: {doc.error}")

// C# - key phrase extraction
using Azure;
using Azure.AI.TextAnalytics;

var endpoint = Environment.GetEnvironmentVariable("LANGUAGE_ENDPOINT");
var key = Environment.GetEnvironmentVariable("LANGUAGE_KEY");
var client = new TextAnalyticsClient(
    new Uri(endpoint), new AzureKeyCredential(key));

var documents = new List<string>
{
    "The new Azure AI services update includes improved "
    + "sentiment analysis and faster key phrase extraction.",
    "Our support team handled over 10,000 tickets last month."
};

var results = await client.ExtractKeyPhrasesBatchAsync(documents);

foreach (var doc in results.Value)
{
    if (doc.HasError) continue;
    Console.WriteLine($"Key phrases: {string.Join(", ", doc.KeyPhrases)}");
}

Entity Recognition

Named Entity Recognition (NER) identifies and categorizes named entities in text. The prebuilt model recognizes entities across several categories including people, organizations, locations, dates and times, quantities, currencies, and URLs. Each detected entity includes a category, subcategory, confidence score, and the character offsets in the original text where the entity appears.

# Python - entity recognition
results = client.recognize_entities(documents)

for i, doc in enumerate(results):
    if not doc.is_error:
        print(f"\nDocument {i+1} entities:")
        for entity in doc.entities:
            print(f"  {entity.text} ({entity.category})"
                  f" [{entity.confidence_score:.2%}]")
    else:
        print(f"Error: {doc.error}")

Sentiment Analysis

Sentiment analysis evaluates the emotional tone of text and classifies it as positive, negative, neutral, or mixed. The API returns both document-level and sentence-level sentiment scores. Each score includes three confidence values — positive, neutral, and negative — that sum to 1.0. The dominant sentiment is the one with the highest confidence score.

Sentence-level analysis is particularly valuable for longer documents where sentiment may shift. For example, a product review might begin with positive statements about delivery speed and end with negative statements about product quality. Document-level sentiment aggregates these into an overall score, but sentence-level analysis reveals the nuance.

results = client.analyze_sentiment(documents)

for i, doc in enumerate(results):
    if not doc.is_error:
        print(f"\nDocument {i+1}: Overall sentiment = {doc.sentiment}")
        print(f"  Confidence: Positive={doc.confidence_scores.positive:.2f}, "
              f"Neutral={doc.confidence_scores.neutral:.2f}, "
              f"Negative={doc.confidence_scores.negative:.2f}")
        for sentence in doc.sentences:
            print(f"  Sentence: \"{sentence.text[:50]}...\"")
            print(f"    Sentiment: {sentence.sentiment}")
    else:
        print(f"Error: {doc.error}")

Sentiment analysis works across multiple languages including English, Spanish, French, German, Italian, Portuguese, Japanese, Korean, Chinese, and Arabic. The model was trained on social media, review, and news data, so it performs best on similar text types. For highly domain-specific language, consider using custom text classification instead.

Language Detection

Language detection identifies the dominant language of a text document. The API returns the detected language name, ISO 639-1 code, and a confidence score. It can detect over 120 languages and variants. If the text is too short or ambiguous, the API may return "Unknown" with a low confidence score. The minimum recommended text length for reliable detection is 100 characters.

# Python - language detection
texts = [
    "Hello, how are you today?",
    "Bonjour, comment allez-vous aujourd'hui?",
    "今日はお元気ですか？"
]

results = client.detect_language(texts)

for i, doc in enumerate(results):
    if not doc.is_error:
        print(f"Text {i+1}: Detected {doc.primary_language.name} "
              f"({doc.primary_language.iso6391_name}) "
              f"[{doc.primary_language.confidence_score:.2%}]")

PII Detection

Personally Identifiable Information (PII) detection identifies sensitive information in text that could identify an individual. This includes email addresses, phone numbers, social security numbers, bank account numbers, credit card numbers, passport numbers, driver's license numbers, IP addresses, and medical information (HIPAA entities). PII detection is a critical tool for data privacy compliance, especially for organizations that process customer data and need to redact sensitive information before analysis or storage.

The API returns detected PII entities with their category, subcategory, confidence score, and character offsets. You can use the offsets to redact or mask the sensitive content. The following example demonstrates PII detection and automatic redaction:

results = client.recognize_pii_entities(documents)

for i, doc in enumerate(results):
    if not doc.is_error:
        print(f"Document {i+1}: Redacted text: {doc.redacted_text}")
        for entity in doc.entities:
            print(f"  Found {entity.category}: {entity.text} "
                  f"(confidence: {entity.confidence_score:.2%})")
    else:
        print(f"Error: {doc.error}")

The redacted_text field returns the original text with all PII entities replaced by asterisks, making it safe for logging or further processing. This feature alone makes PII detection valuable for compliance pipelines. You can filter specific PII categories using the pii_categories parameter if you only care about certain types of sensitive data.

Understanding API Responses

All Language API endpoints return responses in a consistent format. Each document in a batch gets its own result object. If a document causes an error — for example, if it exceeds the maximum character limit or contains unsupported content — the error is reported in that document's result rather than failing the entire batch. Your code must check the is_error property before accessing result fields.

Confidence scores are the model's estimate of how likely the prediction is correct. A score of 0.99 means the model is 99% confident. For most applications, you should set a threshold for accepting results. A common threshold is 0.80 for entity recognition and 0.75 for sentiment analysis. For PII detection, use a higher threshold of 0.90 to minimize false positives that could lead to unnecessary redaction.

Custom Entity Recognition

When the prebuilt entity recognition model does not cover your domain-specific terms, you can train a custom NER model using Azure AI Language Studio. Custom NER allows you to define your own entity types and train a model to recognize them in your documents. Common use cases include extracting medical terms from clinical notes, identifying legal clauses in contracts, and extracting part numbers from technical documentation.

To create a custom NER model, you need a Language resource in a supported region (West Europe or East US for custom features). You define your schema by creating entity types and subtypes, then label documents in Language Studio by selecting text spans and assigning entity types. After labeling, you train and evaluate the model, then deploy it to a prediction endpoint. Custom NER is covered in detail in Chapter 9, which focuses entirely on custom text models.

Batch Processing and Error Handling

The Language SDKs support synchronous and asynchronous batch processing. Synchronous methods accept up to 10 documents per call and are suitable for real-time applications. Asynchronous methods accept larger batches (up to 100,000 documents) and process them as a long-running job, suitable for offline data processing pipelines.

For error handling, check is_error on each document result. Common errors include InvalidDocument (exceeded character limit), UnsupportedLanguageCode (language not supported for the selected feature), and InternalServerError (transient service failure). Implement retry logic with exponential backoff for transient errors. The Azure SDK retry policies handle most transient failures automatically.

1,The product exceeded my expectations. Delivery was fast and the quality is amazing.,en
2,The software crashes frequently and the support team takes days to respond.,en
3,El producto es bueno pero el envío fue muy lento.,es
4,Service correct. Rien d'exceptionnel.,fr

Summary

Azure AI Language provides five core prebuilt NLP capabilities. Key phrase extraction identifies main topics. Entity recognition finds classified entities such as people, organizations, locations, and dates. Sentiment analysis evaluates emotional tone at both document and sentence levels. Language detection identifies the language of text across 120+ languages. PII detection finds and redacts sensitive personal information. All APIs return structured JSON with confidence scores and handle errors at the document level within a batch. Custom NER extends the prebuilt model for domain-specific terms. The next chapter covers custom text classification and custom NER in depth, showing you how to train and deploy your own text models.