Azure Video Indexer is a cloud service that extracts rich metadata from video files using AI. It combines multiple Azure AI services — including Computer Vision, Face API, Speech-to-Text, Text-to-Speech, and Language services — into a single video analysis pipeline. You upload a video, and Video Indexer processes it to produce a comprehensive set of insights organized into a structured JSON output called the video index.
The service is designed for media companies, content platforms, enterprise training libraries, and any organization that needs to search, catalog, or analyze video content at scale. On the AI-102 exam, Video Indexer appears in the "Implement image and video processing" domain alongside Computer Vision and the Face API.
Video Indexer extracts a wide variety of insights from video content. Each insight type corresponds to a specific AI model that runs during the indexing pipeline.
| Insight | Description | Source AI Service |
|---|---|---|
| Face detection | Detects and tracks faces appearing in the video | Face API |
| OCR | Extracts on-screen text from frames | Computer Vision |
| Speech transcription | Converts spoken words to text with timestamps | Speech-to-Text |
| Sentiment analysis | Analyzes emotional tone of speech segments | Language Service |
| Labels | Identifies objects and scenes (e.g., "car," "outdoor," "meeting room") | Computer Vision |
| Keyframes | Identifies representative frames for scene transitions | Computer Vision |
| Scene detection | Segments video into logical scenes and shots | Computer Vision |
| People tracking | Detects and tracks people across frames with position data | Computer Vision |
| Audio effects detection | Detects events like applause, laughter, silence, and speech | Custom audio model |
| Brand detection | Identifies brand logos and names in video | Computer Vision |
| Topic modeling | Derives topics from transcribed speech | Language Service |
| Language identification | Detects the primary spoken language | Language Service |
All insights are correlated along a unified timeline. For example, you can find every frame where a specific person spoke about a specific topic. This temporal correlation is one of Video Indexer's most powerful features and a key differentiator from analyzing video using individual AI services separately.
Video Indexer uses a separate account model distinct from other Azure AI services. You create a Video Indexer account through the Azure portal by searching for "Video Indexer." The account is linked to an Azure Media Services resource and a Storage account. During creation, you select the region, pricing tier, and media reserved units.
There are two account types. A trial account provides limited free indexing minutes (typically 10 hours) and is suitable for evaluation. A paid account offers unlimited indexing with per-minute billing. For production workloads, you need the paid account. The pricing varies by region and the number of media reserved units allocated to the account.
Once the account is created, you access it through the Video Indexer portal at www.videoindexer.ai. The portal provides a web interface for uploading, indexing, browsing, searching, and customizing insights. You also receive an API key for programmatic access, available in the Account Settings section of the portal.
The simplest way to get started is through the Video Indexer portal. Sign in with your Azure account, click "Upload," select a video file, and choose the indexing preset. The default preset analyzes all available insights. You can also choose a basic preset that only transcribes speech, which is faster and consumes fewer indexing minutes.
During indexing, Video Indexer displays a progress bar showing the current stage: uploading, analyzing audio, analyzing video, and generating insights. The time required depends on the video duration, the number of insights selected, and the allocated media reserved units. A 10-minute video typically indexes in 2-5 minutes with default settings.
After indexing completes, the portal displays the video player alongside a timeline of detected insights. You can click on any insight — a face, a spoken phrase, an OCR text detection — and the player jumps to the corresponding timestamp. This interactive exploration is useful for verifying the quality of the indexing before using the API to extract data programmatically.
For automated pipelines, use the Video Indexer REST API. The API requires an access token obtained from the Account API. The workflow involves three steps: obtain an access token, upload the video with indexing parameters, and poll the indexing status until complete.
import requests
import os
import time
account_id = os.environ["VIDEO_INDEXER_ACCOUNT_ID"]
api_key = os.environ["VIDEO_INDEXER_API_KEY"]
# Step 1: Get an access token
token_url = (
f"https://api.videoindexer.ai/auth/{account_id}/Account"
f"?allowEdit=true"
)
headers = {"Ocp-Apim-Subscription-Key": api_key}
response = requests.get(token_url, headers=headers)
access_token = response.json().get("accessToken")
# Step 2: Upload the video and start indexing
upload_url = (
f"https://api.videoindexer.ai/{account_id}/Videos"
f"?accessToken={access_token}"
f"&name=sample-video"
f"&privacy=Private"
f"&language=en-US"
)
with open("meeting_recording.mp4", "rb") as f:
files = {"file": f}
upload_response = requests.post(upload_url, files=files)
video_id = upload_response.json().get("id")
print(f"Video uploaded, ID: {video_id}")
# Step 3: Wait for indexing to complete
state = "Processing"
while state == "Processing":
time.sleep(10)
state_url = (
f"https://api.videoindexer.ai/{account_id}/Videos/{video_id}/Index"
f"?accessToken={access_token}"
)
state_response = requests.get(state_url)
state = state_response.json().get("state")
print(f"Indexing state: {state}")
# Step 4: Get the full index
index = state_response.json()
print(f"Indexing completed. Duration: {index['videos'][0]['duration']} seconds")
# Extract some insights
insights = index["videos"][0]["insights"]
if "faces" in insights:
for face in insights["faces"]:
print(f"Detected person: {face.get('name', 'Unknown')} "
f"appeared {len(face.get('appearances', []))} times")
if "transcript" in insights:
total_words = sum(
len(item.get("words", []))
for item in insights["transcript"]
)
print(f"Total transcribed words: {total_words}")Video Indexer supports several video formats including MP4, MOV, AVI, and WMV. The maximum file size depends on your account tier — typically 2 GB for trial accounts and unlimited for paid accounts. For files larger than 2 GB, you must use the upload from URL option instead of direct upload. The supported frame rates range from 0.5 to 60 FPS, and the minimum resolution is 240p.
Video Indexer allows you to customize several aspects of the indexing pipeline to improve accuracy for your specific content. Brand customization lets you add custom brand names that the model might not recognize by default. You can upload a list of brand names and logos, and Video Indexer will detect them in your videos.
Language customization enables you to improve speech transcription accuracy for domain-specific vocabulary. You can upload a text file containing industry terms, product names, and acronyms. The speech model adapts to recognize these words more accurately. This is particularly valuable for technical content, medical lectures, or any video containing specialized terminology.
Face customization allows you to identify specific people by name. During indexing, Video Indexer detects faces but labels them as "Unknown" by default. You can use the portal to identify unknown faces and assign names. Once labeled, Video Indexer recognizes those individuals in future uploads. The face model is account-specific and does not share data across accounts.
Video Indexer provides embeddable widgets that you can integrate into web applications. There are three main widget types. The Player widget embeds the video player with a synchronized timeline of detected insights. The Insights widget displays the extracted metadata in a searchable, filterable panel. The Editor widget allows users to customize insights by naming faces and correcting transcriptions directly in the browser.
To embed a widget, generate an embed code from the Video Indexer portal's "Embed" section. The embed code includes an iframe pointing to the Video Indexer widget endpoint with the video ID and access token. You can customize the appearance using CSS and configure which insights to display through URL parameters.
Spatial Analysis is a separate Azure AI service under the Computer Vision umbrella that analyzes video streams to understand people's presence, movements, and interactions in physical spaces. Unlike Video Indexer, which analyzes pre-recorded video files, Spatial Analysis is designed for real-time processing of live camera feeds from CCTV cameras, retail stores, factories, or office environments.
Spatial Analysis is deployed as a Docker container that runs on Azure IoT Edge devices or on-premises servers with GPU support. The container processes video frames locally and sends detected events to Azure IoT Hub or a custom endpoint. This on-premises deployment is necessary because real-time video processing at high frame rates requires low latency and high bandwidth that cloud-only solutions cannot provide.
Spatial Analysis provides several prebuilt operations that you configure through a JSON settings file. Each operation focuses on a specific type of spatial understanding.
| Operation | Description | Use Case |
|---|---|---|
| personcrossingline | Detects when a person crosses a virtual line | Entry counting, restricted zone monitoring |
| personcrossingpolygon | Detects when a person enters or exits a defined area | Occupancy tracking, room capacity monitoring |
| persondistance | Measures distance between people in a zone | Social distancing compliance |
| personzone | Detects presence of people in specific zones | Workstation occupancy, queue monitoring |
| personattributes | Detects attributes like person's orientation and speed | Movement analysis, behavior insights |
Deploying Spatial Analysis requires a Docker host with a GPU. The container image is hosted on Microsoft Container Registry and supports NVIDIA GPU acceleration through CUDA. Before deployment, you must configure the container with your Computer Vision resource endpoint and key for billing.
# Pull the Spatial Analysis container
docker pull mcr.microsoft.com/azure-cognitive-services/vision/spatial-analysis:latest
# Run the container with a sample operation configuration
docker run --rm -it \
--gpus all \
-p 5000:5000 \
-e EULA=accept \
-e BILLING=https://your-vision-endpoint.cognitiveservices.azure.com/ \
-e APIKEY=your_vision_key \
mcr.microsoft.com/azure-cognitive-services/vision/spatial-analysis:latest \
python run.py --operations PersonCrossingPolygon \
--config '{"zones": [{"name": "entry", "polygon": [[0,0],[0,1],[1,1],[1,0]]}]}'The container uses a JSON configuration file that defines zones, lines, and camera parameters. You specify which video source to use (RTSP camera URL or local video file), the frames-per-second rate for analysis, the detection confidence threshold, and the output event destination. Events are sent to Azure IoT Hub or can be written to a local file or stdout for testing.
Exam questions often test your ability to choose between Video Indexer and Computer Vision for video scenarios. Use Video Indexer when you need multiple types of insights from the same video — transcriptions, faces, sentiment, labels, OCR, and scene detection — all correlated on a single timeline. Use Computer Vision (with the Read API or Image Analysis) when you need to analyze individual video frames without audio processing, or when you need OCR on static images rather than video. A common exam scenario describes a requirement to "search for spoken keywords in a library of training videos" — the correct answer is Video Indexer because it provides speech transcription and topic modeling.
The Video Indexer output is a JSON document called the video index. It has a hierarchical structure. The top level contains an array of videos (usually one). Each video object has sections for insights (detected faces, transcript, OCR, labels, etc.), summaries (aggregated statistics), and the video metadata including duration, resolution, and processing state.
The insights section is where the detailed analysis lives. The transcript array contains each spoken sentence with start and end times, confidence scores, and individual word-level timing. The faces array lists detected persons with their appearances (which frames they appear in), and a thumbnail ID for each face. The ocr array contains detected text with its on-screen location and timing. The labels array contains objects and scenes with their appearance intervals. The keyframes array lists representative frames with thumbnails.
Each insight type includes a instances array that specifies the time range during which the insight is active. This temporal information is what makes Video Indexer searches powerful — you can find exactly when something occurs in the video and jump to that point.
For accurate transcription, ensure the video has clear audio without excessive background noise. Use a dedicated microphone rather than relying on the camera's built-in microphone. For face detection, ensure subjects are facing the camera and well-lit. Avoid extreme camera angles that obscure facial features. For OCR, ensure on-screen text is large enough and has sufficient contrast against the background.
When indexing large volumes of videos, consider using the parallel upload feature and manage your indexing queue through the API. Video Indexer supports batch processing through the API, but each video is processed independently. Plan your indexing schedule to avoid exceeding your account's concurrent job limits.
Review the generated insights after indexing. The AI models are not perfect and may miss or misinterpret content. The Video Indexer portal provides editing tools to correct transcriptions, name faces, and adjust detected brands. Corrected insights improve future indexing if you enable model adaptation.
Create a Python script called video_insights.py that performs the following tasks:
video_index.json for offline inspection.Open the saved JSON in a code editor and examine the structure. Identify how face appearances are linked to timestamps and how transcript segments connect to speaker IDs. Write a brief summary of the structure you observed.
Azure Video Indexer combines multiple AI services into a single video analysis pipeline that extracts faces, speech transcription, sentiment, labels, OCR, scene detection, and more from video content. All insights are correlated on a unified timeline, making it easy to search and navigate video content. You can customize brands, language models, and face identification to improve accuracy for your specific domain. Spatial Analysis complements Video Indexer by providing real-time analysis of live camera feeds for presence detection, movement tracking, and zone monitoring, deployed as a Docker container on edge devices. The next chapter covers Natural Language Processing with Azure AI Language services, including sentiment analysis, entity recognition, and key phrase extraction.