← Back to Tutorials Chapter 6

Face API: Detection and Recognition

Face Detection vs. Face Recognition

The Azure Face API provides two distinct categories of functionality that are often confused. Face detection locates human faces in images and returns bounding box coordinates. It also extracts facial attributes such as age, emotion, and facial hair. Face detection works on any face it can find in an image and does not require any prior enrollment or training. It is purely a detection and analysis capability.

Face recognition goes a step further. It identifies or verifies faces by comparing them against a database of enrolled faces. Recognition requires a PersonGroup or similar structure where you have already registered and trained known faces. The key distinction for the exam is that detection answers "where are the faces and what do they look like?" while recognition answers "who is this person?"

On the AI-102 exam, you must understand which API operations fall under each category. Detection operations include Face - Detect, Face - Find Similar, and Face - Group. Recognition operations include Face - Verify and Face - Identify. The Face API also supports quality checks such as Face - Detect with returnFaceAttributes to check blur, exposure, and noise levels before sending a face to the recognition pipeline.

Azure Face API Capabilities

The Face API exposes five main operations. Detection locates faces and returns their bounding boxes plus optional attributes. Verification compares two faces and returns a confidence score indicating whether they belong to the same person. Identification matches a detected face against a PersonGroup database to find the best match. Find Similar takes a face and searches a face list for similar-looking faces. Group divides unknown faces into groups based on visual similarity.

Operation	Category	Description	API Endpoint
Face - Detect	Detection	Find faces and extract attributes	/detect
Face - Verify	Recognition	Check if two faces belong to the same person	/verify
Face - Identify	Recognition	Match a face against a PersonGroup	/identify
Face - Find Similar	Detection	Search for similar faces in a face list	/findsimilars
Face - Group	Detection	Divide faces into similarity groups	/group

Face Attributes

When you call the Detect API with the returnFaceAttributes parameter, the service returns detailed information about each detected face. The available attributes cover multiple categories of facial analysis.

Attribute Category	Attributes	Notes
Age	Estimated age in years	Approximate, not a precise measurement
Emotion	Anger, contempt, disgust, fear, happiness, neutral, sadness, surprise	Returned as confidence scores that sum to 1.0
Glasses	NoGlasses, ReadingGlasses, Sunglasses, SwimmingGoggles	Detects the type of eyewear
Facial Hair	Beard, Moustache, Sideburns	Confidence score for each
Makeup	EyeMakeup, LipMakeup	Boolean presence indicators
Accessories	Headwear, Glasses, Mask	Detected accessories with confidence
Blur	BlurLevel: Low, Medium, High	Critical for recognition quality checks
Exposure	ExposureLevel: Good, OverExposure, UnderExposure	Indicates lighting quality
Noise	NoiseLevel: Low, Medium, High	Indicates image grain or compression artifacts
Hair	HairColor, Bald, Invisible	Detects hair color with confidence
Occlusion	ForeheadOccluded, EyeOccluded, MouthOccluded	Indicates if parts of the face are covered
QualityForRecognition	High, Medium, Low	Overall suitability for recognition

The quality attributes — blur, exposure, noise, and qualityForRecognition — are especially important in production systems. The Face API requires high-quality face images for accurate recognition. If a detected face has high blur or low quality, the system should prompt the user to provide a better image rather than attempting recognition on a poor-quality face. The qualityForRecognition attribute simplifies this by providing a single rating of whether the face is suitable for recognition.

Creating a Face API Resource

To use the Face API, you create a Face resource in the Azure portal. Search for "Face" in the marketplace and select the Face service. You can choose between the Free F0 tier (limited transactions) and Standard S0 tier (production scale). After creation, note the endpoint URL and one of the authentication keys from the "Keys and Endpoint" page.

You can also create the resource using Azure CLI:

# Create a Face API resource
az cognitiveservices account create \
    --name face-ai102-study \
    --resource-group rg-ai102-study \
    --kind Face \
    --sku F0 \
    --location westus

# Get the endpoint
az cognitiveservices account show \
    --name face-ai102-study \
    --resource-group rg-ai102-study \
    --query "properties.endpoint"

# List the keys
az cognitiveservices account keys list \
    --name face-ai102-study \
    --resource-group rg-ai102-study

Install the Face SDK for your language of choice. For Python, install the azure-cognitiveservices-vision-face package. For C#, add the Microsoft.Azure.CognitiveServices.Vision.Face NuGet package.

Detecting Faces in an Image

Face detection returns bounding box coordinates and can optionally return facial attributes. The following Python example detects faces in a local image file and prints the attributes:

import os
from azure.cognitiveservices.vision.face import FaceClient
from msrest.authentication import CognitiveServicesCredentials

endpoint = os.environ["FACE_ENDPOINT"]
key = os.environ["FACE_KEY"]

client = FaceClient(endpoint, CognitiveServicesCredentials(key))

with open("group_photo.jpg", "rb") as f:
    image_data = f.read()

attributes = ["age", "emotion", "glasses", "facialHair",
              "makeup", "accessories", "blur", "exposure", "noise"]

detected_faces = client.face.detect_with_stream(
    image_data,
    return_face_attributes=attributes,
    return_face_id=True
)

print(f"Detected {len(detected_faces)} face(s)")
for i, face in enumerate(detected_faces):
    rect = face.face_rectangle
    print(f"\nFace {i+1}:")
    print(f"  Location: ({rect.left}, {rect.top}), {rect.width}x{rect.height}")
    print(f"  Face ID: {face.face_id}")
    att = face.face_attributes
    print(f"  Age: {att.age}")
    print(f"  Glasses: {att.glasses}")
    print(f"  Blur: {att.blur.blur_level} ({att.blur.value:.2f})")
    emotions = att.emotion
    primary_emotion = max(vars(emotions).items(), key=lambda x: x[1])
    print(f"  Primary emotion: {primary_emotion[0]} ({primary_emotion[1]:.2%})")

// C# - detecting faces
using Microsoft.Azure.CognitiveServices.Vision.Face;
using Microsoft.Azure.CognitiveServices.Vision.Face.Models;

var endpoint = Environment.GetEnvironmentVariable("FACE_ENDPOINT");
var key = Environment.GetEnvironmentVariable("FACE_KEY");

var client = new FaceClient(new ApiKeyServiceClientCredentials(key))
{
    Endpoint = endpoint
};

using var imageStream = File.OpenRead("group_photo.jpg");
var attributes = new List
{
    FaceAttributeType.Age, FaceAttributeType.Emotion,
    FaceAttributeType.Glasses, FaceAttributeType.FacialHair,
    FaceAttributeType.Makeup, FaceAttributeType.Accessories,
    FaceAttributeType.Blur, FaceAttributeType.Exposure,
    FaceAttributeType.Noise
};

var faces = await client.Face.DetectWithStreamAsync(
    imageStream,
    returnFaceAttributes: attributes,
    returnFaceId: true
);

Console.WriteLine($"Detected {faces.Count} face(s)");
for (int i = 0; i < faces.Count; i++)
{
    var face = faces[i];
    var rect = face.FaceRectangle;
    Console.WriteLine($"\nFace {i + 1}:");
    Console.WriteLine($"  Location: ({rect.Left}, {rect.Top}), {rect.Width}x{rect.Height}");
    Console.WriteLine($"  Age: {face.FaceAttributes.Age}");
    Console.WriteLine($"  Glasses: {face.FaceAttributes.Glasses}");
}

Face Identification Workflow

Face identification requires a multi-step workflow that involves creating a PersonGroup, enrolling faces, training the group, and then identifying faces against it. The PersonGroup is a container that holds Person objects, each representing one individual. Each Person has a collection of enrolled face images that train the model to recognize that individual.

Step 1: Create a PersonGroup

person_group_id = "employees"
client.person_group.create(person_group_id, "Employees")

Step 2: Create Person objects and add faces

# Create Person entries
alice = client.person_group_person.create(
    person_group_id, "Alice"
)
bob = client.person_group_person.create(
    person_group_id, "Bob"
)

# Add face images for each person
with open("alice_01.jpg", "rb") as f:
    client.person_group_person.add_face_from_stream(
        person_group_id, alice.person_id, f
    )
with open("alice_02.jpg", "rb") as f:
    client.person_group_person.add_face_from_stream(
        person_group_id, alice.person_id, f
    )

with open("bob_01.jpg", "rb") as f:
    client.person_group_person.add_face_from_stream(
        person_group_id, bob.person_id, f
    )

Step 3: Train the PersonGroup

client.person_group.train(person_group_id)

# Wait for training to complete
import time
while True:
    status = client.person_group.get_training_status(
        person_group_id
    )
    if status.status == "succeeded":
        print("Training completed")
        break
    elif status.status == "failed":
        print("Training failed")
        break
    time.sleep(1)

Step 4: Identify a face

# First detect a face in the query image
with open("test_photo.jpg", "rb") as f:
    test_image = f.read()

test_faces = client.face.detect_with_stream(
    test_image, return_face_id=True
)

if test_faces:
    face_ids = [f.face_id for f in test_faces]
    results = client.face.identify(
        face_ids, person_group_id
    )

    for result in results:
        if result.candidates:
            best = result.candidates[0]
            person = client.person_group_person.get(
                person_group_id, best.person_id
            )
            print(f"Identified as: {person.name}")
            print(f"Confidence: {best.confidence:.2%}")
        else:
            print("Face did not match any enrolled person")

Face Verification

Verification checks whether two faces belong to the same person. You pass two face IDs (obtained from the Detect API) and receive a confidence score and a boolean verdict. You also control the confidence threshold — by default, the API uses a recommended threshold, but you can adjust it for stricter or looser matching behavior.

# Detect faces in two images
with open("person_a.jpg", "rb") as f:
    faces_a = client.face.detect_with_stream(f, return_face_id=True)

with open("person_b.jpg", "rb") as f:
    faces_b = client.face.detect_with_stream(f, return_face_id=True)

if faces_a and faces_b:
    verify_result = client.face.verify_face_to_face(
        faces_a[0].face_id,
        faces_b[0].face_id
    )
    print(f"Same person: {verify_result.is_identical}")
    print(f"Confidence: {verify_result.confidence:.2%}")

Ethical Considerations and Responsible AI

Microsoft has implemented stringent responsible AI policies for the Face API. In 2023, Microsoft retired the Emotion and Gender attributes and deprecated the use of the Face API for emotional state inference, gender identification, and age estimation in several regions. The service now includes a Limited Access policy that requires approval for using face recognition features (Identify and Verify). Detection-only features (Detect, Find Similar, Group) remain generally available without special approval.

To use face recognition features, you must apply through Microsoft's Limited Access program. Microsoft reviews each application to ensure the use case aligns with responsible AI principles. Applications must not be used for unlawful discrimination, surveillance without consent, or any purpose that violates human rights.

The key ethical considerations for face recognition include: obtaining explicit consent from individuals before enrolling their faces, providing clear disclosure when face recognition is in use, ensuring the system is accurate across diverse demographics to avoid racial or gender bias, and implementing human oversight to review recognition decisions in high-stakes scenarios.

Best Practices for Face Image Quality

Recognition accuracy depends heavily on image quality. For best results, use images where the face is at least 200x200 pixels, the face is frontal or within 30 degrees of frontal, lighting is even without harsh shadows, the subject has a neutral expression, and there are no obstructions like masks, sunglasses, or heavy scarves covering key facial features. The Face API allows you to check quality programmatically using the qualityForRecognition attribute before attempting identification or verification.

For enrollment, collect images that represent how the person will appear during recognition. If you expect users to be identified in varied lighting conditions, enroll photos captured under those same conditions. Enrolling multiple images of the same person from slightly different angles improves recognition robustness.

Summary

The Face API provides both detection and recognition capabilities. Detection locates faces and extracts attributes such as age, glasses, blur, and accessories. Recognition uses PersonGroups to identify and verify faces against enrolled databases. The identification workflow requires creating a PersonGroup, creating Person entries, enrolling faces, training the group, and then calling Identify. Verification compares two face IDs directly. Responsible AI principles govern the use of face recognition, including Microsoft's Limited Access policy that requires approval for recognition features. Image quality directly impacts accuracy, so always check quality attributes before attempting recognition. The next chapter covers Video Indexer for extracting insights from video content and Spatial Analysis for understanding movement in physical spaces.