← Back to Tutorials Chapter 2

Azure AI Services Overview

What Are Azure AI Services?

Azure AI Services, formerly known as Azure Cognitive Services, are a collection of prebuilt cloud APIs and SDKs that allow applications to incorporate intelligent features without requiring machine learning expertise. These services handle the underlying model training, hosting, and scaling so that you can focus on building application logic. Each service exposes a REST API and provides client SDKs for popular programming languages including C#, Python, Java, JavaScript, and Go.

Microsoft organizes these services into four main categories: Vision, Language, Speech, and Decision. A fifth category, Azure OpenAI, covers generative AI capabilities on a separate infrastructure. In 2023, Microsoft rebranded Cognitive Services to Azure AI Services as part of a broader unification of its AI product portfolio. The underlying APIs and SDKs remain the same; only the naming changed.

Service Categories

Vision Services

Vision services process and analyze visual content. Azure AI Vision provides image analysis features like tagging, object detection, optical character recognition (OCR), description generation, and adult content moderation. Custom Vision allows you to train custom image classifiers and object detectors with your own labeled images. The Face API detects and recognizes human faces in images. Video Indexer extracts insights from video files including transcriptions, faces, spoken keywords, and scene segmentation.

Language Services

Language services analyze and generate text. Azure AI Language includes prebuilt capabilities for sentiment analysis, key phrase extraction, named entity recognition, language detection, and personally identifiable information (PII) detection. Custom text classification and custom named entity recognition allow you to train models on your own data for domain-specific terms. Conversational Language Understanding (CLU) processes natural language inputs to identify intents and extract entities. Custom Question Answering creates knowledge bases from FAQ documents and web pages.

Speech Services

Speech services convert spoken language to text and text to speech. Speech-to-text transcribes audio streams in real time or from recorded files. Text-to-speech produces natural-sounding speech using neural voices. Speech Translation translates spoken language into text or speech in another language. Custom Speech and Custom Voice allow you to train models on domain-specific vocabulary or create unique synthesized voices.

Decision Services

Decision services help applications make informed choices. The Content Safety service detects offensive, inappropriate, or harmful content in text and images. The Anomaly Detector identifies unusual patterns in time-series data. The Personalizer (now deprecated) provided reinforcement-learning-based personalization. For the AI-102 exam, Content Safety is the decision service you should focus on most.

Generative AI

Azure OpenAI Service brings OpenAI's large language models, including GPT-4, GPT-4 Turbo, GPT-4o, and DALL-E, to the Azure platform. These models support text generation, code generation, image generation, embeddings, and function calling. Azure OpenAI is a separate service from Azure AI Services with its own provisioning, pricing, content filtering, and data privacy policies.

Category	Services	Key API
Vision	AI Vision, Custom Vision, Face API, Video Indexer	vision.microsoft.com
Language	AI Language, CLU, QnA Maker	language.cognitiveservices.azure.com
Speech	Speech-to-text, Text-to-speech, Speech Translation	speech.microsoft.com
Decision	Content Safety, Anomaly Detector	contentsafety.cognitiveservices.azure.com
Generative	Azure OpenAI	openai.azure.com

Single-Service vs Multi-Service Resources

When you provision an Azure AI service, you can choose between a single-service resource or a multi-service resource. A single-service resource provides access to exactly one API. For example, creating a "Computer Vision" resource gives you only Vision endpoints. A multi-service resource, called "Cognitive Services" in the portal, provides access to all Azure AI services except Azure OpenAI through a single endpoint and key pair.

Which should you choose? Use a multi-service resource when you are building an application that uses multiple AI services because you manage one key and one endpoint, and you consolidate billing into a single resource. Use a single-service resource when you need isolation for security or cost tracking, or when a service does not support the multi-service option. Some services require single-service resources for specific features like custom training.

Endpoints and Authentication

Every Azure AI service resource has a unique endpoint URL and two authentication keys. The endpoint URL follows the pattern https://<region>.api.cognitive.microsoft.com/ for single-service resources or https://<name>.cognitiveservices.azure.com/ for multi-service resources. You can find both in the Azure portal under "Keys and Endpoint."

Authentication uses one of two methods. The simplest is key-based authentication: include the key in the Ocp-Apim-Subscription-Key HTTP header with every request. The second method uses Azure Active Directory tokens via a managed identity or service principal, which is more secure and recommended for production systems. You can disable key-based authentication entirely and enforce token-based access for stricter security.

Creating an AI Service Resource

You can provision AI services through the Azure portal, Azure CLI, or infrastructure-as-code tools like Bicep and Terraform. The following examples create a Computer Vision resource in the West US region using the Free F0 tier.

Using the Azure Portal

Navigate to the Azure portal, click "Create a resource," search for "Computer Vision," and fill in the required fields: subscription, resource group, region, name, and pricing tier. Click "Review + create" and then "Create." The deployment completes within a minute. After deployment, navigate to the resource and locate the "Keys and Endpoint" page under Resource Management.

Using the Azure CLI

# Create a resource group if you don't have one
az group create --name rg-ai102-study --location westus

# Create a Computer Vision resource
az cognitiveservices account create \
    --name cv-ai102-study \
    --resource-group rg-ai102-study \
    --kind ComputerVision \
    --sku F0 \
    --location westus

# List the keys
az cognitiveservices account keys list \
    --name cv-ai102-study \
    --resource-group rg-ai102-study

# Show the endpoint
az cognitiveservices account show \
    --name cv-ai102-study \
    --resource-group rg-ai102-study \
    --query "properties.endpoint"

Understanding the API Structure

Every Azure AI service follows a consistent API structure. REST endpoints accept JSON payloads and return JSON responses. The general pattern for a request involves an HTTP POST to the service endpoint with the input data in the body and the authentication key in the header.

Here is an example of analyzing an image using the Vision REST API directly:

POST https://westus.api.cognitive.microsoft.com/vision/v3.2/analyze?visualFeatures=Tags,Description
Headers:
  Ocp-Apim-Subscription-Key: YOUR_KEY
  Content-Type: application/json

Body:
{
  "url": "https://example.com/photo.jpg"
}

The response includes a JSON object with the requested visual features. Each feature type returns different fields. For example, the tags field contains an array of tag objects with name and confidence scores, while description contains a list of captions and the tags that generated them.

Using the SDK Clients

The SDKs wrap the REST APIs and provide strongly-typed objects for requests and responses. The following Python example creates a client, calls the analyze endpoint, and prints the caption and tags:

import os
from azure.ai.vision import ImageAnalysisClient
from azure.ai.vision.models import VisualFeatures
from azure.core.credentials import AzureKeyCredential

endpoint = os.environ["VISION_ENDPOINT"]
key = os.environ["VISION_KEY"]

client = ImageAnalysisClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(key)
)

with open("photo.jpg", "rb") as f:
    image_data = f.read()

result = client.analyze(
    image_data=image_data,
    visual_features=[VisualFeatures.TAGS, VisualFeatures.CAPTION]
)

if result.caption:
    print(f"Caption: {result.caption.text} ({result.caption.confidence:.2f})")

if result.tags:
    for tag in result.tags:
        print(f"{tag.name}: {tag.confidence:.2f}")

The equivalent code in C# uses the Azure.AI.Vision packages available as NuGet packages:

using Azure;
using Azure.AI.Vision;
using Azure.AI.Vision.ImageAnalysis;

var endpoint = Environment.GetEnvironmentVariable("VISION_ENDPOINT");
var key = Environment.GetEnvironmentVariable("VISION_KEY");

var client = new ImageAnalysisClient(
    new Uri(endpoint),
    new AzureKeyCredential(key));

using var imageStream = File.OpenRead("photo.jpg");
var result = client.Analyze(
    imageStream,
    VisualFeatures.Tags | VisualFeatures.Caption);

if (result.Value.Caption != null)
{
    Console.WriteLine($"Caption: {result.Value.Caption.Text} ({result.Value.Caption.Confidence:F2})");
}

if (result.Value.Tags != null)
{
    foreach (var tag in result.Value.Tags)
    {
        Console.WriteLine($"{tag.Name}: {tag.Confidence:F2}");
    }
}

Container Deployment for AI Services

Azure AI services are available as Docker containers that you can deploy to any Docker-compatible environment, including on-premises servers, edge devices, or other cloud platforms. Containers are useful when you need to run AI services in disconnected environments, process data locally to meet compliance requirements, reduce latency for real-time applications, or handle high-volume workloads without network round trips.

To use a container, you pull the image from Microsoft Container Registry (mcr.microsoft.com), configure it with your service's endpoint and key for billing, and run it on your Docker host. The container periodically sends usage metrics to Azure for billing but processes all requests locally.

# Pull the Vision Read container image
docker pull mcr.microsoft.com/azure-cognitive-services/vision/read:3.2

# Run the container
docker run --rm -it -p 5000:5000 \
    -e EULA=accept \
    -e BILLING_ENDPOINT=https://westus.api.cognitive.microsoft.com/ \
    -e APIKEY=YOUR_KEY \
    mcr.microsoft.com/azure-cognitive-services/vision/read:3.2

Once running, you call the container at http://localhost:5000 using the same REST API as the cloud service. Containers are available for Vision Read, Face, Text Analytics (Language), Speech-to-text, Text-to-speech, Form Recognizer (Document Intelligence), and others. Not all services offer container support, and some containers require specific licensing agreements.

Monitoring and Diagnostic Logging

Every Azure AI service resource emits metrics and logs that you can monitor through Azure Monitor. The key metrics include total calls, failed calls, data in and data out, and latency (response time). You can create metric alerts that fire when, for example, the failure rate exceeds a threshold or the number of calls approaches the rate limit. Setting up monitoring is essential for production applications to detect issues before they affect users.

To enable diagnostic logging, navigate to the "Diagnostic settings" page of your AI service resource, click "Add diagnostic setting," select the log categories you want to collect (usually Audit and AllMetrics), and send them to a Log Analytics workspace, storage account, or Event Hub. The Audit log records every API call with details including the caller's IP address, the operation performed, the HTTP status code, and the response size.

Managing Costs

The Free F0 tier provides a limited number of transactions per month at no charge. For the Computer Vision service, F0 gives 20 transactions per minute and 5,000 transactions per month. The Standard S0 tier charges per transaction and has higher rate limits. You can also purchase commitment tiers for predictable pricing at a discount. Commitment tiers let you reserve a specific throughput for a monthly fee, which reduces the per-transaction cost by up to 40% compared to pay-as-you-go.

For exam scenarios, you should know that F0 is suitable for development and testing but not production. Choose S0 when you need higher throughput. Choose commitment tiers when you have consistent, predictable traffic and want to reduce costs.

Virtual Network Security

For production environments, you should restrict network access to your AI services. Azure AI services support virtual network integration that limits traffic to only those originating from your selected virtual networks. You can also configure IP firewall rules to allow traffic from specific IP ranges while blocking all other traffic. When you enable a virtual network, you must also enable a service endpoint or private endpoint for the service. Private endpoints assign a private IP address from your virtual network to the service, keeping traffic within the Microsoft Azure backbone network rather than traversing the public internet.

Managing Authentication with RBAC

Instead of using keys, you can authenticate to Azure AI services using Azure Active Directory (Azure AD) identities. This approach uses Role-Based Access Control (RBAC) to assign permissions. The built-in roles for Cognitive Services include Cognitive Services Contributor (full access), Cognitive Services User (read access to keys and endpoints), and Cognitive Services Metrics Reader (read-only access to metrics). Assign these roles to managed identities, service principals, or user accounts through the Azure portal or CLI.

Responsible AI Principles

Microsoft has established six core principles for responsible AI that guide the development and deployment of AI systems. Understanding these principles is important both for the exam and for real-world practice. The AI-102 exam asks scenario questions where you must identify which principle applies to a given concern.

• Fairness: AI systems should treat all people fairly and avoid reinforcing biases. This means testing models for disparate impact across demographic groups.
• Reliability and Safety: AI systems should perform reliably under expected conditions and fail gracefully when encountering unexpected inputs. Rigorous testing and monitoring support this principle.
• Privacy and Security: AI systems must protect user data and respect privacy. This includes data encryption, access controls, and compliance with data protection regulations.
• Inclusiveness: AI systems should empower everyone and be accessible to people of all abilities. Designing for a diverse user base from the start is key.
• Transparency: Users should understand how AI systems make decisions. This includes providing explanations, documentation, and disclosure when AI is involved.
• Accountability: Organizations should take responsibility for how their AI systems operate. This requires governance structures, impact assessments, and human oversight mechanisms.

Microsoft provides tools to help implement these principles, including Fairlearn for assessing fairness, the InterpretML toolkit for model interpretability, and the Responsible AI dashboard in Azure Machine Learning. The AI-102 exam may ask you to identify which principle is most relevant to a given scenario or recommend a mitigating action for a specific concern.

Summary

Azure AI Services provide prebuilt intelligence across Vision, Language, Speech, Decision, and Generative AI categories. You provision these services as single-service or multi-service resources, access them via REST APIs or SDKs, and secure them with keys or Azure AD authentication. Containers allow offline or edge deployment. Monitoring, cost management, virtual network security, and responsible AI practices are all essential considerations for production use. Mastering these fundamentals is the foundation for every subsequent chapter in this book.