← Back to Tutorials Chapter 5

Custom Vision: Classification and Detection

When to Use Custom Vision

Azure AI Vision's prebuilt models recognize thousands of common objects, scenes, and concepts. However, many real-world applications require identifying specialized items that the prebuilt models have not been trained on. Custom Vision fills this gap by allowing you to train your own image classifiers and object detectors using your labeled images.

Custom Vision is the right choice when your problem involves domain-specific categories like manufacturing defects, retail product variants, medical conditions in X-rays, specific animal breeds, vehicle models, or proprietary inventory items. It is also appropriate when you need to distinguish between subtle visual differences that generic models would group together, such as different grades of lumber or types of fabric texture.

Do not use Custom Vision when the prebuilt Vision API already covers your needs, when you need real-time processing at very high throughput without managing model training, or when you have fewer than 15-20 images per category, because the model will not generalize well with so little data.

Image Classification vs. Object Detection

Custom Vision supports two primary project types. Image classification assigns a single label or multiple labels to the entire image. Object detection identifies the location of objects within an image using bounding boxes and can detect multiple instances of the same or different objects in a single image.

Choose image classification when you need to categorize an entire scene or image. Examples include sorting photos by location type (beach, mountain, city), grading product quality (pass, fail), or identifying which plant species appears in a photo. Image classification is faster to label because you only assign tags to each image rather than drawing bounding boxes around objects.

Choose object detection when the location of the object matters or when multiple objects of different types appear in the same image. Examples include counting vehicles in a parking lot, detecting defects at specific positions on a manufactured part, locating text regions in a scanned document, or identifying multiple animal species in a wildlife camera image. Object detection requires more effort to label but provides richer information.

Characteristic	Image Classification	Object Detection
Output	Label(s) for the entire image	Labels + bounding boxes per object
Labeling effort	Tags per image	Bounding boxes per object per image
Use case	Scene categorization, quality grading	Object counting, spatial analysis
Multi-object images	One label if multiclass, multiple if multilabel	Multiple labels with positions
Training time	Faster, simpler	Slower, more computationally intensive

Creating a Custom Vision Service Resource

Custom Vision is a separate Azure service with its own resource type. You can create a Custom Vision resource in the Azure portal by searching for "Custom Vision" and selecting it from the marketplace. Unlike other AI services, Custom Vision has two separate endpoints: one for training and one for prediction. Each endpoint can have its own resource with its own pricing tier.

For development and learning, create a Free F0 training resource (2 training hours per month) and a Free F0 prediction resource (5,000 predictions per month). For production, use Standard S0 for both training and prediction. S0 has no hourly training limits and provides higher throughput for predictions.

# Create Custom Vision training resource
az cognitiveservices account create \
    --name cv-training-ai102 \
    --resource-group rg-ai102-study \
    --kind CustomVision.Training \
    --sku F0 \
    --location westus

# Create Custom Vision prediction resource
az cognitiveservices account create \
    --name cv-prediction-ai102 \
    --resource-group rg-ai102-study \
    --kind CustomVision.Prediction \
    --sku F0 \
    --location westus

# List training keys
az cognitiveservices account keys list \
    --name cv-training-ai102 \
    --resource-group rg-ai102-study

# List prediction keys
az cognitiveservices account keys list \
    --name cv-prediction-ai102 \
    --resource-group rg-ai102-study

Labeling Images for Training

The quality of your training data directly determines the quality of your model. Custom Vision expects a minimum of 15 images per tag for classification and 30 images per tag for object detection, though more images almost always improve results. Images should represent the full range of variations the model will encounter in production: different lighting conditions, angles, backgrounds, sizes, orientations, and occlusion levels.

For image classification, upload images to the Custom Vision portal and assign tags to each image. You can tag individual images or select multiple images and apply the same tag to all of them. The portal supports drag-and-drop uploads and bulk tagging.

For object detection, upload images and draw bounding boxes around each object instance. The portal provides a visual editor where you draw rectangles on the image and assign a tag to each rectangle. Be precise with bounding boxes — include the entire object but minimize the background area inside the box. Overlapping objects should each have their own bounding box.

For the exam, remember that you can also import labeled images through the Custom Vision SDK, which is useful for automating the labeling pipeline when you have existing annotations in formats like COCO JSON or Pascal VOC XML.

Training a Classification Model in the Portal

The Custom Vision portal at customvision.ai provides a graphical interface for managing projects, uploading images, tagging, training, and evaluating models. To train a classification model, create a new project, select "Classification" as the project type, choose either "Multiclass" or "Multilabel," and upload your tagged images. Click the "Train" button to start training with either "Quick Training" (faster, less accurate) or "Advanced Training" (slower, more accurate, billed in training hours).

The portal displays training progress and, upon completion, shows the evaluation results including precision, recall, and average precision metrics. You can also perform a quick test by uploading an image the model has not seen before and reviewing the predicted tags and confidence scores.

Training a Classification Model with the Python SDK

For automated training pipelines, use the Custom Vision SDK. The following Python example creates a project, uploads images, tags them, and trains a classification model:

import os
from azure.cognitiveservices.vision.customvision.training import CustomVisionTrainingClient
from azure.cognitiveservices.vision.customvision.training.models import ImageFileCreateEntry, Tag
from msrest.authentication import ApiKeyCredentials

training_key = os.environ["CUSTOMVISION_TRAINING_KEY"]
training_endpoint = os.environ["CUSTOMVISION_TRAINING_ENDPOINT"]
prediction_key = os.environ["CUSTOMVISION_PREDICTION_KEY"]
prediction_resource_id = os.environ["CUSTOMVISION_PREDICTION_RESOURCE_ID"]

# Authenticate
credentials = ApiKeyCredentials(in_headers={"Training-key": training_key})
trainer = CustomVisionTrainingClient(training_endpoint, credentials)

# Create a new project
project = trainer.create_project("Fruit Classifier")

# Create tags
tag_apple = trainer.create_tag(project.id, "Apple")
tag_banana = trainer.create_tag(project.id, "Banana")
tag_orange = trainer.create_tag(project.id, "Orange")

# Upload images
base_dir = "training_images"
image_list = []

for tag_name, tag_id in [("Apple", tag_apple.id), ("Banana", tag_banana.id), ("Orange", tag_orange.id)]:
    tag_dir = os.path.join(base_dir, tag_name)
    for filename in os.listdir(tag_dir):
        with open(os.path.join(tag_dir, filename), "rb") as f:
            image_bytes = f.read()
        image_list.append(
            ImageFileCreateEntry(name=filename, contents=image_bytes, tag_ids=[tag_id])
        )

# Upload in batches of 64
for i in range(0, len(image_list), 64):
    batch = image_list[i:i+64]
    upload_result = trainer.create_images_from_files(project.id, images=batch)
    print(f"Uploaded batch {i//64 + 1}: {len(batch)} images")

# Train the model
print("Training started...")
iteration = trainer.train_project(project.id, training_type="Advanced")
while iteration.status != "Completed":
    iteration = trainer.get_iteration(project.id, iteration.id)
    print(f"Training status: {iteration.status}")

print(f"Training completed. Iteration ID: {iteration.id}")

# Publish the model to the prediction endpoint
trainer.publish_iteration(
    project.id,
    iteration.id,
    "fruit-classifier-model",
    prediction_resource_id
)
print("Model published to prediction endpoint.")

Training an Object Detection Model in the Portal

Object detection training in the portal follows the same general workflow as classification, with one critical difference: you must draw bounding boxes around each object. After uploading images to a new project set to "Object Detection" type, click on each image and drag rectangles around the objects you want to detect. Assign a tag to each bounding box. The portal displays the bounding box coordinates as you draw.

After tagging, click "Train" and select "Advanced Training" for object detection. Training takes longer than classification because the model must learn both object identity and spatial location. The evaluation metrics include mean Average Precision (mAP), which is the primary metric for object detection quality.

Training an Object Detection Model with the SDK

For object detection, you must provide region information (bounding box coordinates) when uploading images. The region is specified as left, top, width, and height, each normalized to the range [0, 1] relative to the image dimensions.

from azure.cognitiveservices.vision.customvision.training.models import (
    ImageFileCreateEntry, Region, ImageUrlCreateEntry
)

# For object detection, include region with each tagged image
image_entries = []

# Example: upload an image with a bounding box for "Dog"
with open("dog_photo.jpg", "rb") as f:
    image_bytes = f.read()

region = Region(
    tag_id=tag_dog.id,
    left=0.1,    # 10% from left edge
    top=0.2,     # 20% from top edge
    width=0.7,   # 70% of image width
    height=0.6   # 60% of image height
)

entry = ImageFileCreateEntry(
    name="dog_001.jpg",
    contents=image_bytes,
    regions=[region]
)
image_entries.append(entry)

# Upload and train
trainer.create_images_from_files(project.id, images=image_entries)

iteration = trainer.train_project(project.id, training_type="Advanced")
while iteration.status != "Completed":
    iteration = trainer.get_iteration(project.id, iteration.id)
    time.sleep(1)

trainer.publish_iteration(project.id, iteration.id, "od-model", prediction_resource_id)

Evaluating Model Metrics

After training, Custom Vision provides evaluation metrics on a held-out test set (automatically selected from your uploaded images). The key metrics depend on the project type.

For classification, the primary metrics are precision and recall. Precision (also called accuracy) measures how many of the model's predictions were correct: TP / (TP + FP). Recall measures how many of the actual positive cases the model found: TP / (TP + FN). The F1 score is the harmonic mean of precision and recall. The portal also shows a per-tag precision and recall breakdown.

For object detection, the primary metric is mean Average Precision (mAP), computed at an intersection-over-union (IoU) threshold of 0.50 (or 0.75 for stricter evaluation). mAP@0.50 measures how well the predicted bounding boxes overlap with the ground truth boxes, averaged across all object classes. A higher mAP indicates more accurate localization. The exam does not require you to calculate these metrics, but you should understand what they represent and how to interpret them for model improvement.

Metric	Classification	Object Detection	Interpretation
Precision	Yes	Per class	How many predicted positives are correct
Recall	Yes	Per class	How many actual positives were found
F1 Score	Yes	Per class	Harmonic mean of precision and recall
mAP	No	Yes	Mean Average Precision across classes
AP per class	No	Yes	Average Precision for a single class

Publishing and Consuming a Custom Vision Model

After training, you must publish an iteration before it can be called through the prediction API. Publishing assigns a published name to the iteration and makes it available at a fixed prediction endpoint URL. You can publish multiple iterations under different names for A/B testing or staged rollouts.

To call the prediction endpoint, use the Custom Vision prediction SDK with the prediction key and endpoint URL. The following example shows how to call a published classification model:

# Python - calling a published classification model
from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
from msrest.authentication import ApiKeyCredentials

prediction_key = os.environ["CUSTOMVISION_PREDICTION_KEY"]
prediction_endpoint = os.environ["CUSTOMVISION_PREDICTION_ENDPOINT"]
project_id = "your-project-id"
published_name = "fruit-classifier-model"

credentials = ApiKeyCredentials(in_headers={"Prediction-key": prediction_key})
predictor = CustomVisionPredictionClient(prediction_endpoint, credentials)

with open("test_fruit.jpg", "rb") as f:
    image_data = f.read()

results = predictor.classify_image(project_id, published_name, image_data)

for prediction in results.predictions:
    print(f"{prediction.tag_name}: {prediction.probability:.2%}")

// C# - calling a published object detection model
using Microsoft.Azure.CognitiveServices.Vision.CustomVision.Prediction;
using Microsoft.Azure.CognitiveServices.Vision.CustomVision.Prediction.Models;

var predictionKey = Environment.GetEnvironmentVariable("CUSTOMVISION_PREDICTION_KEY");
var predictionEndpoint = Environment.GetEnvironmentVariable("CUSTOMVISION_PREDICTION_ENDPOINT");
var projectId = Guid.Parse("your-project-id");
var publishedName = "od-model";

var credentials = new ApiKeyServiceClientCredentials(predictionKey);
var client = new CustomVisionPredictionClient(credentials)
{
    Endpoint = predictionEndpoint
};

using var imageStream = File.OpenRead("test_dog.jpg");
var result = client.DetectImage(projectId, publishedName, imageStream);

foreach (var prediction in result.Predictions)
{
    var box = prediction.BoundingBox;
    Console.WriteLine($"Detected: {prediction.TagName} " +
        $"({prediction.Probability:P2}) " +
        $"at [{box.Left:F3}, {box.Top:F3}, {box.Width:F3}, {box.Height:F3}]");
}

Exporting Models for Offline Use

Custom Vision supports exporting trained models to several formats for execution at the edge or in offline environments. Supported export formats include TensorFlow (saved model and frozen graph), ONNX (for Windows ML or ONNX Runtime), CoreML (for Apple devices), Dockerfile (for container deployment), and VAIDK (for Vision AI Development Kit).

To export a model, select the iteration in the Custom Vision portal and click "Export." Then choose the target platform. The exported model includes the model architecture, trained weights, and a label file. You load the model using the corresponding framework and pass images for inference without making network calls.

On the exam, remember that export is only available for certain project types and domains. Compact domains (compact classification and compact object detection) support exporting. General (non-compact) domains do not. When a scenario requires offline inference or edge deployment, you must select a compact domain during project creation.

Tips for Building Quality Training Datasets

The success of your Custom Vision model depends almost entirely on the quality and diversity of your training data. Follow these guidelines to maximize model accuracy.

Collect images that represent the full range of production conditions. If your model will classify products on a conveyor belt, include images taken under factory lighting, different conveyor belt speeds (motion blur), and different camera angles. Do not rely on web images alone — capture real images from the deployment environment.

Use negative samples. For classification, include images that do not belong to any of your target tags. Upload them without any tag, and the model learns to produce low confidence when it encounters unfamiliar content. For object detection, include images without any objects — the model should learn not to produce false positives on empty backgrounds.

Apply data augmentation. Custom Vision automatically applies basic augmentation during training, but you can improve results by manually augmenting your dataset with rotations (up to 30 degrees), horizontal flips, brightness adjustments, and random crops. The Python Imaging Library (Pillow) is useful for this.

Avoid ambiguous or inconsistent tagging. If a single training image could be labeled with multiple tags and you are using multiclass classification, the model will be confused. Ensure that each image clearly belongs to its assigned tag. For object detection, ensure bounding boxes are precise and consistent in size relative to the object.

Monitor overfitting. If your training metrics are very high but the model performs poorly on new images, you are overfitting. Solutions include adding more training data, applying stronger regularization, or reducing model complexity by switching to a compact domain. Early stopping is built into Custom Vision training, so you do not need to manage it yourself.

Custom Vision Pitfalls to Avoid

Several common mistakes lead to poor Custom Vision models. Using too few images per tag is the most frequent issue. The portal will train with as few as 5 images, but the model will not generalize. Using images that are too similar to each other within a tag also hurts — if all your muffin photos are on a white plate, the model might learn to recognize "white plate" instead of "muffin." Include diverse backgrounds.

Another mistake is training with images that contain multiple taggable objects for a multiclass project. If a photo contains both a croissant and a muffin, you should either exclude it or switch to multilabel. For object detection, failing to tag all instances of an object in an image teaches the model that those instances are negative examples, which directly harms recall.

Finally, remember that the prediction endpoint is separate from the training endpoint. You must create a prediction resource, publish the iteration, and use the prediction key in your application code. Many developers forget this step and receive 403 Forbidden errors when trying to call the prediction API with the training key.

Summary

Custom Vision lets you train custom image classifiers and object detectors using your own labeled images. Classification assigns tags to entire images, while object detection provides both tags and bounding boxes for individual objects within an image. Training requires a balanced dataset with a minimum of 15-30 images per tag. After training, you evaluate the model using precision, recall, and mAP metrics, publish the iteration, and consume it through the prediction API. Models can be exported to TensorFlow, ONNX, CoreML, and Docker for offline or edge deployment. The quality of your training data determines the quality of your model — invest effort in collecting diverse, well-labeled images. The next chapter covers the Face API for facial detection and recognition.