DALL-E 3 is the image generation model available through Azure OpenAI. It creates high-quality images from natural language descriptions, supporting styles from photorealistic to cartoon and oil painting. The API accepts a text prompt and returns one or more generated images as URLs or base64-encoded data. DALL-E 3 also supports image editing through inpainting — you provide an existing image with a transparent mask region, and the model fills in the masked area based on a text description. This allows you to replace objects, add elements, or extend backgrounds in existing images.
Prompt design for DALL-E follows different rules than text generation. Be specific about the subject, style, medium, lighting, color palette, composition, and mood. Instead of "a cat," use "a photorealistic orange tabby cat sitting on a wooden desk in a sunlit room, shallow depth of field, warm tones, 8K resolution." The model handles negative prompts poorly — instead of saying "no people," describe what you want without mentioning people. Each generated image varies slightly due to the model's stochastic nature; if you need consistency, include a seed parameter or generate multiple images and select the best match.
# Python - DALL-E image generation
import os
from openai import AzureOpenAI
client = AzureOpenAI(
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
api_key=os.environ["AZURE_OPENAI_KEY"],
api_version="2024-10-21"
)
response = client.images.generate(
model="dall-e-3",
prompt="A photorealistic orange tabby cat sitting on a wooden desk "
"in a sunlit modern office, shallow depth of field, warm morning light",
n=1,
size="1024x1024",
quality="hd",
style="vivid"
)
image_url = response.data[0].url
revised_prompt = response.data[0].revised_prompt
print(f"Generated image URL: {image_url}")
print(f"Revised prompt: {revised_prompt}")
# Download and save the image
import requests
img_data = requests.get(image_url).content
with open("generated_image.png", "wb") as f:
f.write(img_data)
print("Image saved as generated_image.png")The API response includes a revised_prompt field — this is the prompt that DALL-E actually used after rewriting your input for safety and quality. Microsoft automatically revises prompts to remove harmful content and to optimize image quality. You can compare the original and revised prompts to understand how DALL-E interprets your description. The revised prompt is also useful as a prompt engineering learning tool.
Function calling allows GPT models to detect when they should call external functions and return structured JSON arguments for those calls. Instead of generating unstructured text, the model outputs a function name and a JSON object with the parameters, which your application parses and executes. This enables AI-powered features like order placement, database queries, calendar management, and API orchestration — all driven by natural language.
To use function calling, you define one or more tools (formerly called functions) in the API request. Each tool has a name, description, and a JSON Schema defining its parameters. When the model determines that a function should be called, it returns a response with finish_reason="tool_calls" instead of "stop". Your application executes the function with the provided arguments and sends the result back as a tool response message. The model then generates a final response incorporating the function result.
# Python - function calling for order management
import os
import json
from openai import AzureOpenAI
client = AzureOpenAI(
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
api_key=os.environ["AZURE_OPENAI_KEY"],
api_version="2024-10-21"
)
# Define tools that the model can call
tools = [
{
"type": "function",
"function": {
"name": "get_order_status",
"description": "Get the status of a customer order by order ID",
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"description": "The order ID (e.g., ORD-12345)"
}
},
"required": ["order_id"]
}
}
},
{
"type": "function",
"function": {
"name": "cancel_order",
"description": "Cancel an existing order",
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"description": "The order ID to cancel"
},
"reason": {
"type": "string",
"description": "Reason for cancellation"
}
},
"required": ["order_id", "reason"]
}
}
}
]
# Simulated function implementations
def get_order_status(order_id):
statuses = {
"ORD-1001": "Shipped",
"ORD-1002": "Processing",
"ORD-1003": "Delivered"
}
return json.dumps({"order_id": order_id,
"status": statuses.get(order_id, "Not Found")})
def cancel_order(order_id, reason):
return json.dumps({"order_id": order_id,
"status": "Cancelled",
"reason": reason})
messages = [
{"role": "system", "content": "You are a helpful order management assistant."},
{"role": "user", "content": "What is the status of order ORD-1001?"}
]
response = client.chat.completions.create(
model="gpt-4o-deployment",
messages=messages,
tools=tools,
tool_choice="auto"
)
choice = response.choices[0]
message = choice.message
if choice.finish_reason == "tool_calls":
for tool_call in message.tool_calls:
func_name = tool_call.function.name
args = json.loads(tool_call.function.arguments)
if func_name == "get_order_status":
result = get_order_status(**args)
elif func_name == "cancel_order":
result = cancel_order(**args)
messages.append(message)
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": result
})
# Second call with function results
response = client.chat.completions.create(
model="gpt-4o-deployment",
messages=messages,
tools=tools
)
print(response.choices[0].message.content)
else:
print(message.content)Embeddings are vector representations of text that capture semantic meaning. Azure OpenAI provides the text-embedding-3-large and text-embedding-3-small models, which convert text strings into arrays of floating-point numbers (vectors). Similar texts produce similar vectors — the cosine similarity between two embedding vectors indicates their semantic similarity. Embeddings are the foundation of semantic search, clustering, recommendation systems, and anomaly detection.
To use embeddings, you send a text string to the embeddings API and receive a vector (default dimension 3072 for large, 1536 for small). You can reduce dimensions by setting the dimensions parameter — the model supports dimensional reduction from 3072 down to any size, with the smaller size trading accuracy for storage and compute efficiency. After generating embeddings, you store them in a vector database (like Azure AI Search with vector support or Cosmos DB with vector search) for similarity-based retrieval.
# Python - generating and using embeddings
from openai import AzureOpenAI
import numpy as np
client = AzureOpenAI(
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
api_key=os.environ["AZURE_OPENAI_KEY"],
api_version="2024-10-21"
)
def get_embedding(text: str) -> list[float]:
response = client.embeddings.create(
model="text-embedding-3-large",
input=text,
dimensions=1536 # reduced from 3072 for efficiency
)
return response.data[0].embedding
# Generate embeddings for documents
documents = [
"Azure AI Search provides full-text search and AI enrichment",
"Document Intelligence extracts structured data from forms",
"Content Safety moderates text and images for harmful content",
"Computer Vision analyzes images for objects and text"
]
doc_embeddings = [get_embedding(doc) for doc in documents]
# Search query
query = "How do I moderate harmful user content?"
query_embedding = get_embedding(query)
# Compute cosine similarity
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
similarities = [
(doc, cosine_similarity(query_embedding, emb))
for doc, emb in zip(documents, doc_embeddings)
]
similarities.sort(key=lambda x: x[1], reverse=True)
print("Top matching documents:")
for doc, score in similarities:
print(f" {score:.4f}: {doc}")Retrieval Augmented Generation (RAG) is an architecture that combines a retrieval system (search) with a generation system (GPT). Instead of relying solely on the model's training data, RAG retrieves relevant documents from a knowledge base and includes them in the prompt as context. This grounds the model's answers in factual, up-to-date information and eliminates hallucination for the covered topics. RAG is the standard approach for building knowledge-base Q&A systems, customer support chatbots, and enterprise search assistants.
The RAG workflow has four steps. First, indexing — chunk your documents into segments (typically 512-1024 tokens each) and compute embeddings for each chunk. Store the chunks and their embeddings in a vector database. Second, retrieval — when a user asks a question, compute the query's embedding and perform a vector similarity search to find the most relevant chunks. Third, augmentation — insert the retrieved chunks into a prompt template alongside the user's question. Fourth, generation — send the augmented prompt to GPT and return the answer.
# Python - RAG pattern with Azure AI Search + Azure OpenAI
import os
from openai import AzureOpenAI
from azure.search.documents import SearchClient
from azure.core.credentials import AzureKeyCredential
openai_client = AzureOpenAI(
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
api_key=os.environ["AZURE_OPENAI_KEY"],
api_version="2024-10-21"
)
search_client = SearchClient(
endpoint=os.environ["AI_SEARCH_ENDPOINT"],
index_name="knowledge-base",
credential=AzureKeyCredential(os.environ["AI_SEARCH_KEY"])
)
def rag_answer(question: str) -> str:
# Step 1: Generate embedding for the question
query_embedding = openai_client.embeddings.create(
model="text-embedding-3-small",
input=question
).data[0].embedding
# Step 2: Retrieve relevant documents via vector search
search_results = search_client.search(
search_text=None,
vector_queries=[{
"kind": "vector",
"vector": query_embedding,
"fields": "content_vector",
"k": 3
}],
select=["content", "title"],
top=3
)
# Step 3: Build the augmented prompt
context = "\n\n".join([
f"Source: {doc['title']}\n{doc['content']}"
for doc in search_results
])
system_message = (
"You are an Azure AI expert assistant. Answer the user's question "
"based on the provided context. If the context does not contain "
"enough information to answer, say 'I cannot find that information "
"in the available sources.' Cite the source title when referencing "
"specific information."
)
messages = [
{"role": "system", "content": system_message},
{"role": "user", "content": f"Context:\n{context}\n\n"
f"Question: {question}"}
]
# Step 4: Generate the answer
response = openai_client.chat.completions.create(
model="gpt-4o-deployment",
messages=messages,
temperature=0.3,
max_tokens=800
)
return response.choices[0].message.content
print(rag_answer("What is the content filtering process in Azure Content Safety?"))Fine-tuning creates a customized version of a GPT model by training it on your own dataset. Unlike RAG, which retrieves external context at inference time, fine-tuning embeds knowledge and behavior patterns into the model weights during training. This makes the model inherently better at your specific task — it learns the vocabulary, writing style, domain knowledge, and response structure from your training data. Fine-tuning is appropriate when you have a consistent, well-defined task with sufficient high-quality training data (at least hundreds of examples).
The fine-tuning process involves preparing training data in the chat completions format — a JSONL file where each line is a conversation with system, user, and assistant messages. You upload the file to Azure OpenAI, submit a fine-tuning job, and wait for the job to complete. The resulting model is deployed as a custom model deployment with its own endpoint. Fine-tuning costs are based on training tokens — you pay for the tokens processed during training in addition to the base model's inference costs. Fine-tuned models inherit the base model's context window but may have reduced ability on unrelated tasks (catastrophic forgetting), so only fine-tune for the specific task.
# Preparing fine-tuning data (JSONL format)
training_examples = [
{
"messages": [
{"role": "system", "content": "Classify customer support "
"tickets into categories: Billing, Technical, Account, Other."},
{"role": "user", "content": "My credit card was charged twice "
"for the same subscription."},
{"role": "assistant", "content": "Billing"}
]
},
{
"messages": [
{"role": "system", "content": "Classify customer support "
"tickets into categories: Billing, Technical, Account, Other."},
{"role": "user", "content": "I cannot log in to my account "
"after changing my password."},
{"role": "assistant", "content": "Account"}
]
}
]
import json
with open("training_data.jsonl", "w") as f:
for example in training_examples:
f.write(json.dumps(example) + "\n")
# Submit fine-tuning job
client = AzureOpenAI(
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
api_key=os.environ["AZURE_OPENAI_KEY"],
api_version="2024-10-21"
)
response = client.fine_tuning.jobs.create(
training_file=client.files.create(
file=open("training_data.jsonl", "rb"),
purpose="fine-tune"
).id,
model="gpt-35-turbo",
suffix="ticket-classifier"
)
job_id = response.id
print(f"Fine-tuning job submitted: {job_id}")
# Check job status
import time
while True:
job = client.fine_tuning.jobs.retrieve(job_id)
print(f"Status: {job.status}")
if job.status in ["succeeded", "failed", "cancelled"]:
break
time.sleep(30)
print(f"Fine-tuned model: {job.fine_tuned_model}")Build a complete RAG system that answers questions about Azure AI services using Azure AI Search as the vector store and GPT-4o for generation.
The AI-102 exam frequently tests the difference between RAG and fine-tuning. Use RAG when you need to ground answers in specific, changing, or large-scale knowledge bases — it does not require training, is easy to update (just update the index), and the model always has access to the latest information. Use fine-tuning when you need to teach the model a consistent behavior pattern, output format, or domain vocabulary — it requires training data but produces faster inference (no retrieval step) and can learn nuances that RAG cannot capture. You can combine both — fine-tune for behavior and use RAG for knowledge. For embeddings model selection, use text-embedding-3-large (3072 dimensions) for maximum accuracy, text-embedding-3-small (1536 dimensions) for cost-sensitive applications, and consider the dimensions parameter to reduce storage costs while retaining most of the accuracy. For token optimization, chunk documents strategically — smaller chunks (256-512 tokens) improve retrieval precision while larger chunks (512-1024 tokens) provide more context for generation. Include document titles and metadata in the retrieved chunks so the model can cite sources. Use the max_tokens parameter to limit response length and avoid exceeding the model's context window when many chunks are retrieved.
Advanced Azure OpenAI patterns extend beyond basic chat completions. DALL-E generates images from text descriptions with support for inpainting and style control. Function calling enables structured API orchestration — the model outputs function names and arguments that your application executes. Embeddings convert text to semantic vectors for similarity search, clustering, and retrieval. RAG combines vector search with GPT generation to produce grounded, factual answers based on a knowledge base. Fine-tuning customizes model behavior through training on domain-specific data. Each pattern has distinct strengths — RAG for knowledge grounding, fine-tuning for behavioral consistency, function calling for automation, and embeddings for semantic understanding. On the AI-102 exam, focus on choosing between RAG and fine-tuning, understanding embeddings model tradeoffs, and optimizing token usage across all patterns. The next chapter covers the Bot Framework SDK for building conversational AI applications.