← Back to Tutorials Chapter 12

Speech-to-Text

Azure Speech-to-Text Overview: Real-Time vs. Batch

Azure Speech-to-Text converts spoken language into written text. It is one of the most mature Azure AI services, used in applications ranging from real-time captioning and voice commands to call center transcription and meeting notes generation. The service supports two primary modes: real-time transcription and batch transcription. Real-time transcription streams audio to the API and returns partial and final results as the speech is happening. The latency is typically under one second for final results. This mode is ideal for live captioning, voice assistants, dictation, and any scenario where a user is waiting for a response.

Batch transcription processes prerecorded audio files asynchronously. You submit the audio file via a REST endpoint, and the service transcribes it in the background. You poll the job status and retrieve the results when complete. Batch mode is designed for large volumes of audio — hours of call center recordings, meeting archives, lecture recordings, or podcast episodes. It is more cost-efficient than real-time for large files because you can submit multiple files simultaneously and the pricing is per audio hour rather than per second of processed audio.

Feature	Real-Time	Batch
Latency	Sub-second (streaming)	Minutes to hours (async)
Input	Microphone stream or audio file	Audio file URI (Azure Blob Storage)
Use case	Live captioning, voice assistants	Call centers, meeting archives
Partial results	Yes (streaming)	No (only final)
API	Speech SDK streaming	REST API
Pricing	Per second of audio	Per hour of audio (lower rate)

Real-Time Transcription: Microphone and Audio File Input

Real-time transcription uses the Azure Speech SDK, which manages audio capture, streaming, and result handling. For microphone input, the SDK uses the device's default microphone. You create a SpeechRecognizer, start continuous recognition, and handle events as they fire. The Recognizing event fires for partial results while the user is still speaking, and the Recognized event fires when a complete utterance has been transcribed. Between these two events, the service may revise its transcription multiple times as more audio context becomes available.

For file input, you use the same SpeechRecognizer but pass an AudioConfig created from a WAV file path. The SDK reads the file at real-time speed and generates transcription events as if it were live audio. This is useful for testing transcription quality on prerecorded audio without setting up a batch job. You can also use push or pull audio streams for custom stream handling scenarios.

Batch Transcription: Asynchronous Processing of Audio Files

Batch transcription is the right choice when you need to transcribe large volumes of prerecorded audio without real-time constraints. The process begins by uploading your audio files to an Azure Blob Storage container that the Speech service can access. You then create a batch transcription job via the REST API, providing the URIs of the audio files and optionally specifying the locale, model, and output format. The service processes the files and writes the transcription results to a designated container or returns them inline when you fetch the job.

Batch transcription supports the same base models as real-time, plus custom speech models. It can handle multiple audio formats including WAV, MP3, FLAC, OGG, and OPUS. The results include the full transcription with word-level timestamps, speaker diarization if enabled, and confidence scores for each word and phrase. You can submit up to 1000 files per job, making it suitable for enterprise-scale transcription workloads.

Custom Speech Models: Training on Domain-Specific Vocabulary

The base Speech-to-Text models perform well on general conversation, news reading, and dictated text. However, they struggle with domain-specific vocabulary — medical terminology, legal jargon, product names, technical acronyms, and names of people or places. Custom speech models solve this by training the base model on your own data. You provide text data (sentences and phrases representative of your domain) and optionally audio data with matching transcriptions.

Custom speech training starts with creating a Custom Speech project in Speech Studio at speech.microsoft.com. You upload text data as plain text files or with pronunciation markings. The training process adapts the base model's language model to favor your domain vocabulary. If you also provide audio with transcriptions, the acoustic model is adapted to the speaking styles in your audio (accents, background noise patterns, speaking rate). The result is a custom model that you can deploy as a custom endpoint. Custom endpoints cost the same as standard endpoints but provide significantly higher accuracy on your domain content.

Language and Accent Adaptation

Azure Speech-to-Text supports over 100 languages and variants. For languages with multiple regional variants (English, Spanish, French, Arabic), you specify the locale code when creating the recognizer. The service uses a locale-specific model tuned to that variant's pronunciation and vocabulary. You do not need to train custom models for different accents of the same language — the base model handles common accents within each locale. However, for strong regional accents or dialects not represented in the base model, custom speech training with accent-specific audio can improve accuracy.

Language identification is a separate feature that lets you transcribe audio containing multiple languages without specifying the language in advance. The service detects the language dynamically and switches the transcription model as the speaker changes languages. This is useful for multilingual meetings and customer service calls where the language may change mid-conversation. Language identification adds a small latency overhead and is available only in real-time mode.

Keyword Recognition: Detecting Specific Trigger Words

Keyword recognition lets you detect specific words or short phrases in an audio stream without performing full transcription. This is used for wake-word detection ("Hey Computer," "Alexa," "Start Recording") and for triggering specific actions when a key phrase is spoken. The keyword is defined as a model file that contains the audio characteristics of the target word or phrase. The Speech SDK can recognize keywords locally on the device without sending audio to the cloud, enabling low-latency wake-word detection even in offline scenarios.

To implement keyword recognition, you create a keyword model file in Speech Studio by speaking the target word or phrase multiple times. The generated model file is downloaded and loaded into the Speech SDK's KeywordRecognizer. When the keyword is detected, the recognizer raises an event. You can then start a full speech recognition session to capture the command that follows the wake word. This two-stage approach (keyword detection on-device, then cloud transcription for the actual command) is the standard pattern for voice-activated applications.

Code Sample: Real-Time Speech Transcription

import os
import azure.cognitiveservices.speech as speechsdk

speech_key = os.environ["SPEECH_KEY"]
service_region = os.environ["SPEECH_REGION"]

# Real-time from microphone
speech_config = speechsdk.SpeechConfig(
    subscription=speech_key,
    region=service_region
)
speech_config.speech_recognition_language = "en-US"

# Pull audio from the default microphone
audio_config = speechsdk.audio.AudioConfig(use_default_microphone=True)
recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=audio_config
)

# Handle partial (interim) results
def recognizing(evt):
    print(f"Partial: {evt.result.text}")

# Handle final results
def recognized(evt):
    result = evt.result
    if result.reason == speechsdk.ResultReason.RecognizedSpeech:
        print(f"Final: {result.text}")
        print(f"Lexical: {result.lexical_form}")
        print(f"Confidence: {result.confidence}")
    elif result.reason == speechsdk.ResultReason.NoMatch:
        print(f"No match: {result.no_match_details}")

# Connect event handlers
recognizer.recognizing.connect(recognizing)
recognizer.recognized.connect(recognized)
recognizer.session_started.connect(
    lambda evt: print("Session started"))
recognizer.session_stopped.connect(
    lambda evt: print("Session stopped"))

print("Speak into your microphone. Press Ctrl+C to stop.")
recognizer.start_continuous_recognition()

try:
    import time
    while True:
        time.sleep(0.5)
except KeyboardInterrupt:
    recognizer.stop_continuous_recognition()
    recognizer.close()

// C# - real-time speech from microphone
using Microsoft.CognitiveServices.Speech;
using Microsoft.CognitiveServices.Speech.Audio;

var speechKey = Environment.GetEnvironmentVariable("SPEECH_KEY");
var serviceRegion = Environment.GetEnvironmentVariable("SPEECH_REGION");

var speechConfig = SpeechConfig.FromSubscription(speechKey, serviceRegion);
speechConfig.SpeechRecognitionLanguage = "en-US";

using var audioConfig = AudioConfig.FromDefaultMicrophoneInput();
using var recognizer = new SpeechRecognizer(speechConfig, audioConfig);

recognizer.Recognizing += (s, e) =>
{
    Console.WriteLine($"Partial: {e.Result.Text}");
};

recognizer.Recognized += (s, e) =>
{
    if (e.Result.Reason == ResultReason.RecognizedSpeech)
    {
        Console.WriteLine($"Final: {e.Result.Text}");
        Console.WriteLine($"Confidence: {e.Result.Confidence}");
    }
};

Console.WriteLine("Speak into your microphone. Press Enter to stop.");
await recognizer.StartContinuousRecognitionAsync();
Console.ReadLine();
await recognizer.StopContinuousRecognitionAsync();

Code Sample: Batch Transcription Job

import os
import requests
import time
import json

key = os.environ["SPEECH_KEY"]
region = os.environ["SPEECH_REGION"]
base_url = f"https://{region}.api.cognitive.microsoft.com"
headers = {
    "Ocp-Apim-Subscription-Key": key,
    "Content-Type": "application/json"
}

# Step 1: Create a batch transcription
audio_url = "https://mystorage.blob.core.windows.net/audio/meeting01.wav"
transcription_definition = {
    "contentUrls": [audio_url],
    "locale": "en-US",
    "displayName": "Meeting Transcription 01",
    "properties": {
        "diarizationEnabled": True,
        "wordLevelTimestampsEnabled": True,
        "timeToLive": "PT24H"
    }
}

response = requests.post(
    f"{base_url}/speechtotext/v3.1/transcriptions",
    headers=headers,
    json=transcription_definition
)
transcription_url = response.headers["location"]
transcription_id = transcription_url.split("/")[-1]
print(f"Created transcription: {transcription_id}")

# Step 2: Poll until complete
while True:
    response = requests.get(
        f"{transcription_url}",
        headers=headers
    )
    status = response.json()["status"]
    print(f"Status: {status}")
    if status in ("Succeeded", "Failed"):
        break
    time.sleep(30)

# Step 3: Retrieve results
if status == "Succeeded":
    files_url = f"{transcription_url}/files"
    files_response = requests.get(files_url, headers=headers)
    for file_info in files_response.json()["values"]:
        if file_info["kind"] == "Transcription":
            result_url = file_info["links"]["contentUrl"]
            result = requests.get(result_url)
            transcription = result.json()
            for phrase in transcription["recognizedPhrases"]:
                print(f"Speaker {phrase['speaker']}: "
                      f"{phrase['nBest'][0]['display']}")

Understanding Transcription Responses

The transcription response contains several forms of the transcribed text and metadata. The display text is the formatted version suitable for human reading — it includes capitalization, punctuation, and inverted text normalization (converting "four fifty" to "$4.50" or "four o'clock" to "4:00"). The lexical form is the raw word-by-word transcription with no formatting — everything is lowercase with no punctuation. The NBest list contains alternative transcriptions ranked by confidence, with the top entry being the service's best guess. Each NBest entry includes the display text, lexical form, and the confidence score between 0.0 and 1.0.

Confidence scores indicate the service's certainty about the transcription. Scores above 0.8 are generally reliable. Scores between 0.5 and 0.8 indicate some uncertainty — the word or phrase may be misheard. Scores below 0.5 should be flagged for human review in critical applications like medical transcription. The confidence score is at the phrase level, not the individual word level, unless you enable word-level confidence in the API options.

Handling Noise, Multiple Speakers, and Different Audio Formats

Real-world audio is rarely a clean recording of a single voice. Background noise, multiple speakers talking over each other, poor microphone placement, and varying audio formats all degrade transcription accuracy. The Speech service provides several features to handle these challenges. For noise, the SDK includes audio processing options that apply noise suppression, echo cancellation, and gain adjustment before sending audio to the cloud. Enable these by configuring the AudioProcessingOptions with the DICTATION flag or the specific processing modes you need.

For multiple speakers, enable conversation transcription (also called meeting transcription). This feature uses speaker diarization to assign each utterance to a specific speaker. The service can distinguish up to 10 speakers in a conversation. You provide the audio and the service returns each phrase tagged with a speaker ID. In batch transcription, diarization is enabled via a property flag. In real-time, you use the ConversationTranscriber class instead of SpeechRecognizer and provide a conversation ID.

Audio format requirements are strict. The Speech service expects audio at 16 kHz sample rate, 16-bit depth, mono channel, in WAV or PCM format for best accuracy. Lower sample rates reduce accuracy. Stereo audio should be converted to mono before sending. MP3 and other compressed formats are supported but at reduced accuracy. For batch transcription, the service handles format conversion automatically, but for real-time, ensure your audio source meets the recommended specifications.

Summary

Azure Speech-to-Text transcribes audio into text in real-time or batch mode. Real-time transcription streams audio from a microphone or file and delivers partial and final results via SDK events. Batch transcription processes prerecorded audio files asynchronously and is ideal for large volumes of content. Custom speech models improve accuracy on domain-specific vocabulary by training on text and optionally audio data. Keyword recognition enables on-device wake-word detection. Transcription responses include display text, lexical form, NBest alternatives, and confidence scores. For the AI-102 exam, focus on choosing between real-time and batch for given scenarios, understanding the custom speech training workflow, and knowing the audio format requirements for optimal accuracy. The next chapter covers Text-to-Speech and SSML for generating natural-sounding speech from text.