Azure AI Vision is a cloud-based service that extracts information from images. It provides prebuilt models for a wide range of visual recognition tasks, eliminating the need to train custom models for common scenarios. The service accepts image input as a URL or raw binary data and returns structured JSON containing the analysis results. You can use it to tag images with objects and concepts, generate human-readable captions, detect specific objects and their bounding boxes, identify brands and landmarks, read printed and handwritten text, generate smart cropped thumbnails, and moderate adult or racy content.
The service is part of the Azure AI Services portfolio and is available through a dedicated Computer Vision resource or through the multi-service Cognitive Services resource. The REST API supports version 3.2 (stable) and version 4.0 (preview), with the 4.0 version introducing the unified Image Analysis API that combines multiple capabilities into a single call.
On the AI-102 exam, Computer Vision questions appear in the "Implement image and video processing" domain. You should understand which visual features are available, how to interpret the API responses, and when to use the newer Image Analysis 4.0 API versus the legacy 3.2 endpoints.
The Image Analysis API provides several categories of visual features that you request through the visualFeatures parameter. Each feature returns a specific section in the response JSON.
| Feature | Description | API Parameter |
|---|---|---|
| Tags | Content tags based on objects, actions, and scenes | Tags |
| Caption | A human-readable sentence describing the image | Caption |
| Dense Captions | Captions for individual regions of the image | DenseCaptions |
| Objects | Detected objects with bounding boxes | Objects |
| Brands | Detected brand logos with bounding boxes | Brands |
| Celebrities | Identified celebrities (legacy domain-specific model) | Celebrities |
| Landmarks | Identified landmarks (legacy domain-specific model) | Landmarks |
| Adult | Adult, racy, and gory content classification | Adult |
| Categories | Image categorization using a taxonomy of 86 categories | Categories |
| Color | Dominant colors, accent color, and color scheme analysis | Color |
| Image Type | Indicates if the image is clip art or line drawing | ImageType |
| Faces | Detects faces with age and gender (limited) | Faces |
In the 4.0 Image Analysis API, these features are grouped and invoked through a unified endpoint. The newer API also adds support for people detection and background removal, though these may have additional pricing.
The Image Analysis API returns a structured JSON response. The following example shows the key parts of a response when requesting Tags and Caption features:
{
"captionResult": {
"text": "A group of people standing on a beach at sunset",
"confidence": 0.92
},
"tagsResult": {
"values": [
{ "name": "beach", "confidence": 0.99 },
{ "name": "sunset", "confidence": 0.98 },
{ "name": "people", "confidence": 0.95 },
{ "name": "ocean", "confidence": 0.93 },
{ "name": "sand", "confidence": 0.90 },
{ "name": "standing", "confidence": 0.87 },
{ "name": "group", "confidence": 0.82 },
{ "name": "outdoor", "confidence": 0.80 }
]
},
"metadata": {
"width": 1920,
"height": 1080,
"format": "Jpeg"
},
"modelVersion": "2024-02-01"
}Each tag includes a name and confidence score between 0 and 1. Higher confidence indicates the model is more certain that the tag applies to the image. The caption provides a natural language description. The dense captions feature extends this by providing region-specific captions, each with a bounding box and confidence score.
The celebrity and landmark detection features are considered legacy domain-specific models. They were included in earlier API versions but are not being actively enhanced. For new projects, use the general tags feature and map specific names yourself, or consider using Custom Vision for domain-specific recognition tasks.
Azure AI Vision provides two OCR approaches: the legacy OCR API and the newer Read API. The Read API is the recommended approach for extracting text from images. It supports both printed and handwritten text, handles multiple languages, and processes images with varying orientations and angles.
The Read API uses an asynchronous operation pattern. You submit an image for analysis, receive an operation location URL, and poll that URL until the analysis completes. This is necessary because OCR for complex documents can take several seconds. The synchronous approach is available only for the legacy OCR API, which has limited capabilities compared to Read.
The async workflow involves three steps. First, call the POST /vision/v3.2/read/analyze endpoint with the image URL or binary data. The response includes an Operation-Location header containing the result URL. Second, poll the result URL using GET requests at one-second intervals. Third, when the response status changes to "succeeded," parse the JSON response to extract the recognized text, lines, and words.
# Python - OCR extraction with Read API
import os
import time
from azure.ai.vision import ImageAnalysisClient
from azure.core.credentials import AzureKeyCredential
endpoint = os.environ["VISION_ENDPOINT"]
key = os.environ["VISION_KEY"]
client = ImageAnalysisClient(
endpoint=endpoint,
credential=AzureKeyCredential(key)
)
# Read from a local image file
with open("receipt.jpg", "rb") as f:
image_data = f.read()
# Start the async OCR operation
poller = client.begin_read(image_data)
result = poller.result()
# Process the results
if result.blocks:
for block in result.blocks:
for line in block.lines:
print(f"Line: '{line.text}'")
for word in line.words:
print(f" Word: '{word.text}' (confidence: {word.confidence:.2f})")// C# - OCR extraction with Read API
using Azure;
using Azure.AI.Vision;
using Azure.AI.Vision.ImageAnalysis;
var endpoint = Environment.GetEnvironmentVariable("VISION_ENDPOINT");
var key = Environment.GetEnvironmentVariable("VISION_KEY");
var client = new ImageAnalysisClient(
new Uri(endpoint),
new AzureKeyCredential(key));
using var imageStream = File.OpenRead("receipt.jpg");
var result = client.Read(imageStream);
foreach (var block in result.Value.Blocks)
{
foreach (var line in block.Lines)
{
Console.WriteLine($"Line: '{line.Text}'");
foreach (var word in line.Words)
{
Console.WriteLine($" Word: '{word.Text}' (confidence: {word.Confidence:F2})");
}
}
}The AI-102 exam tests your ability to distinguish between Azure AI Vision's Read API and Azure AI Document Intelligence (formerly Form Recognizer). Use the Read API when you need to extract text from general images, such as signs, menus, or photographs containing text. Use Document Intelligence when you need to extract structured information from forms, invoices, receipts, or identity documents, because it can interpret field relationships and table structures. When the scenario involves form fields or key-value pairs, Document Intelligence is the correct choice.
The smart cropping feature generates thumbnails that preserve the region of interest in an image. Instead of simple resizing that might cut out the subject, the Vision service analyzes the image to identify the most important region and crops around it. You specify the desired output dimensions (width and height), and the service returns a cropped image centered on the region of interest.
The API accepts a smartCropping parameter set to true to enable intelligent cropping. Without this flag, the service performs standard center cropping. Smart cropping is particularly useful for generating thumbnail previews in galleries, profile pictures, and responsive layouts where the subject must remain visible regardless of the aspect ratio.
# Python - smart thumbnail generation
import os
from azure.ai.vision import ImageAnalysisClient
from azure.core.credentials import AzureKeyCredential
endpoint = os.environ["VISION_ENDPOINT"]
key = os.environ["VISION_KEY"]
client = ImageAnalysisClient(
endpoint=endpoint,
credential=AzureKeyCredential(key)
)
with open("photo.jpg", "rb") as f:
image_data = f.read()
# Generate a 200x200 smart-cropped thumbnail
result = client.generate_thumbnail(
image_data=image_data,
width=200,
height=200,
smart_cropping=True
)
with open("thumbnail.jpg", "wb") as f:
f.write(result.thumbnail_data)
print("Thumbnail saved as thumbnail.jpg")Adult content moderation classifies images into three categories: adult content (explicit sexual content), racy content (sexually suggestive content), and gory content (violent or gruesome imagery). The API returns an isAdultContent, isRacyContent, and isGoryContent boolean flag, along with confidence scores that indicate the model's certainty.
This feature is useful for user-generated content platforms, social media applications, image libraries, and any system where users upload images. You can combine the Vision moderation results with Azure AI Content Safety for text-based moderation to create a comprehensive content safety pipeline.
For child safety compliance and to meet Microsoft's responsible AI requirements, applications that process images should always include adult content moderation. The moderation scores are not perfect and should be treated as a guide rather than an absolute classification. You may need to tune thresholds based on your specific content policies.
The Computer Vision service offers two pricing tiers. The Free F0 tier provides 5,000 transactions per month with a rate limit of 20 transactions per minute. The Standard S0 tier charges per 1,000 transactions, with pricing varying by feature. The rate limits for S0 are substantially higher — typically 100 transactions per minute, adjustable by requesting a quota increase through Azure support.
| Operation | F0 Limit | S0 Limit | S0 Price (per 1K) |
|---|---|---|---|
| Analyze Image (all features) | 20/min, 5K/mo | 100/min | ~$1.50 |
| OCR Read | 20/min, 5K/mo | 100/min | ~$1.50 |
| Generate Thumbnail | 20/min, 5K/mo | 100/min | ~$0.80 |
| Detect Objects | 20/min, 5K/mo | 100/min | ~$1.50 |
Monitor your quota usage through the Metrics blade in the Azure portal. Create a metric alert for the "Total Calls" metric with a threshold at 80% of your quota to receive early warnings. For S0 tiers, the rate limit can be increased through a support ticket, but Microsoft reserves the right to deny requests that would negatively impact service stability.
Object detection identifies specific objects in an image and returns bounding box coordinates for each detection. The service recognizes a wide range of object categories including vehicles, animals, furniture, electronics, food, and household items. Each detection includes the object name, confidence score, and a bounding box specified as x, y, width, and height coordinates relative to the image dimensions.
Unlike image tagging, which answers "what is in this image," object detection answers "where is each object in this image." The bounding box coordinates let you draw rectangles around detected objects, enabling applications like inventory management, retail analytics, and automated quality inspection.
Prebuilt object detection is best for common objects. For domain-specific objects such as industrial parts, medical instruments, or specialized equipment, use Custom Vision object detection (covered in Chapter 5) to train a model with your own labeled images.
The following complete Python console application demonstrates image analysis with multiple features, OCR extraction, and thumbnail generation in a single workflow:
import os
import sys
import json
from azure.ai.vision import ImageAnalysisClient
from azure.ai.vision.models import VisualFeatures
from azure.core.credentials import AzureKeyCredential
def analyze_image(client, image_path):
with open(image_path, "rb") as f:
data = f.read()
features = [
VisualFeatures.CAPTION,
VisualFeatures.TAGS,
VisualFeatures.OBJECTS,
VisualFeatures.ADULT
]
result = client.analyze(
image_data=data,
visual_features=features
)
return result
def print_analysis(result):
if result.caption:
print(f"\nCaption: {result.caption.text}")
print(f"Confidence: {result.caption.confidence:.2%}")
if result.tags:
print(f"\nTags ({len(result.tags)}):")
for tag in result.tags:
print(f" {tag.name}: {tag.confidence:.2%}")
if result.objects:
print(f"\nObjects ({len(result.objects)}):")
for obj in result.objects:
box = obj.bounding_box
print(f" {obj.object_property} at ({box.x}, {box.y}), {box.w}x{box.h}")
if result.adult:
print(f"\nAdult Content: {result.adult.is_adult_content}")
print(f"Racy Content: {result.adult.is_racy_content}")
print(f"Gore Content: {result.adult.is_gory_content}")
if __name__ == "__main__":
endpoint = os.environ.get("VISION_ENDPOINT")
key = os.environ.get("VISION_KEY")
if not endpoint or not key:
print("Error: Set VISION_ENDPOINT and VISION_KEY environment variables.")
sys.exit(1)
if len(sys.argv) < 2:
print("Usage: python analyze.py <image_path>")
sys.exit(1)
client = ImageAnalysisClient(
endpoint=endpoint,
credential=AzureKeyCredential(key)
)
image_path = sys.argv[1]
if not os.path.exists(image_path):
print(f"Error: File '{image_path}' not found.")
sys.exit(1)
result = analyze_image(client, image_path)
print_analysis(result)Create a new Python file named image_analyzer.py that performs the following tasks using the Azure AI Vision SDK:
pip install Pillow) and save the result as output.jpg.Run your script against at least three different images: a landscape photo, a photo with multiple people, and a photo containing text. Observe how the caption changes based on the image content and note any tags that seem incorrect or surprising.
To get the best results from Azure AI Vision, follow these guidelines. Provide high-quality images with good lighting and clear subjects. The minimum image size is 50x50 pixels and the maximum is 20,480x20,480 pixels. For OCR, use images where text is at least 10 pixels tall and has good contrast against the background. Images larger than 4 MB take longer to process and may cause timeout errors; consider resizing large images before sending them.
When you need to analyze many images, consider batching. The SDK does not provide a built-in batch API, but you can implement your own batching with parallel requests subject to the rate limit. Use the asynchronous Read API for OCR to avoid blocking your application thread during long-running operations. Cache analysis results when the same images are analyzed multiple times to reduce cost and latency.
The prebuilt Vision models have limited knowledge of specialized domains. They will not recognize proprietary product SKUs, industry-specific equipment, or niche categories. For these requirements, train a Custom Vision model. Also note that the Face detection within the Vision API (under the Faces feature) is limited compared to the dedicated Face API and should not be used for identity verification scenarios.
At the time of writing, Azure AI Vision has two active API versions. Version 3.2 is the stable, long-supported version that most production applications use. Version 4.0 introduces the combined Image Analysis API with a simpler interface and new features like dense captions and background removal.
| Capability | API 3.2 | API 4.0 |
|---|---|---|
| Image analysis | Separate endpoints per feature | Unified endpoint |
| OCR | Read API (async) | Read API (async) |
| Dense captions | Not available | Supported |
| Background removal | Not available | Supported (preview) |
| People detection | Not available | Supported |
| SDK support | Azure.AI.Vision 1.x | Azure.AI.Vision 2.x (preview) |
For the exam, version 3.2 is the current tested version. However, Microsoft updates exam content as services evolve, so check the exam study guide for the specific API version covered at your testing time. The concepts are the same across versions; only the endpoint URLs and response structures differ.
Azure AI Vision provides powerful prebuilt image analysis capabilities including tagging, caption generation, object detection, OCR, thumbnail generation, and content moderation. The Read API handles printed and handwritten text extraction using an asynchronous pattern. Smart cropping produces thumbnails that preserve the image's region of interest. Pricing tiers range from free (F0) with limited throughput to standard (S0) with higher limits and per-transaction billing. Understanding the API response structure and the difference between prebuilt Vision and custom Vision is essential for the exam. In the next chapter, you will learn how to train custom classification and object detection models using Custom Vision.