Custom Text Classification and NER

When to Use Custom Text Models vs. Prebuilt

The prebuilt Azure AI Language APIs handle general-purpose NLP tasks well. They recognize common entities like people and locations, detect sentiment across standard text types, and extract topical key phrases. However, every domain has specialized vocabulary and concepts that prebuilt models do not understand. A medical record contains diagnosis codes and drug names. A legal contract references specific statutes and clauses. A support ticket system uses internal product categories and issue types.

Custom text models fill this gap. You train them on your own labeled documents to recognize the entities and classifications that matter in your domain. Use custom models when the prebuilt APIs do not recognize your domain-specific terms, when you need to classify documents into categories unique to your organization, or when you need higher accuracy than the general-purpose models provide for your specific content. Do not use custom models when the prebuilt APIs already meet your accuracy requirements, when you lack sufficient labeled training data (minimum 10-15 documents per class), or when you need a quick proof-of-concept without investing in data labeling.

Custom Text Classification: Single-Label vs. Multi-Label

Custom text classification assigns one or more categories to a document. The choice between single-label and multi-label depends on whether your categories are mutually exclusive. Single-label classification assigns exactly one category to each document. This is appropriate for scenarios like routing support tickets to a single department, classifying news articles into one primary topic, or grading documents as pass or fail.

Multi-label classification assigns zero, one, or multiple categories to each document. This is appropriate when a document could belong to several categories simultaneously. For example, a product review might discuss both "battery life" and "screen quality." A legal document might involve multiple practice areas. A medical record might have multiple diagnoses. Multi-label models are more flexible but require more training data because the model must learn which combinations of labels are valid.

Custom NER: Entity Extraction for Domain-Specific Terms

Custom NER (Named Entity Recognition) lets you define entity types relevant to your domain and train a model to extract them from text. Unlike classification, which categorizes whole documents, NER extracts specific spans of text and labels them with entity types. This is valuable for extracting structured data from unstructured documents.

Common custom NER use cases include extracting medication names and dosages from clinical notes, identifying contract parties and effective dates from legal agreements, extracting product names and error codes from support tickets, pulling invoice numbers and vendor names from procurement documents, and identifying equipment serial numbers and maintenance dates from service logs.

Creating a Custom Language Project in Language Studio

Azure AI Language Studio provides a graphical interface for creating, training, and deploying custom text models. To get started, navigate to language.cognitive.azure.com and sign in with your Azure credentials. Click on "Custom text classification" or "Custom named entity recognition" to create a new project.

Before creating a project, ensure you have a Language resource in a region that supports custom features. As of the current release, supported regions include West Europe, East US, and South Central US. The resource must be the Standard S0 tier; the Free F0 tier does not support custom model training.

When you create a project in Language Studio, you provide a project name, description, the primary language of your documents, and the Azure Language resource to use. Language Studio then creates a storage container in your associated Azure Storage account where training data and model artifacts are stored.

# Create a custom language project using the Azure CLI
# (after setting up the required storage and RBAC permissions)
az resource create \
    --resource-group rg-ai102-study \
    --resource-type Microsoft.CognitiveServices/accounts \
    --name language-ai102-custom \
    --location eastus \
    --kind TextAnalytics \
    --sku S \
    --yes

Labeling Data for Custom Models

Data labeling is the most critical step in building custom NLP models. The quality of your labels directly determines model accuracy. For custom text classification, you upload documents in Language Studio and assign one or more classes to each document. You can upload documents in two formats: individual files (one document per file) or a single dataset file with all documents and labels in a JSONL format. The JSONL format is recommended for large datasets because it bundles everything into one manageable file.

For custom NER, labeling involves selecting text spans within each document and assigning entity types. Language Studio provides a visual labeling interface where you highlight text with your cursor and select the entity type from a dropdown. Each entity span can overlap with others, and you can assign multiple entity types to the same span if needed.

Best practices for labeling include: labeling all instances of an entity type consistently across all documents, including enough context in each label span to be unambiguous but not including unnecessary surrounding text, balancing the number of labeled instances across entity types, and having at least 10-20 labeled instances per entity type for NER and 10-15 documents per class for classification. Add edge cases to your training data — documents that are borderline between two classes or that contain entities in unusual contexts.

Training and Evaluating Custom Models

After labeling your data, you start training in Language Studio. The training process splits your labeled data into training, validation, and test sets. The default split allocates 80% for training, 10% for validation, and 10% for testing. You can customize this split if you have specific requirements.

Training time depends on the dataset size and the model complexity. A small dataset of 100 documents might train in under a minute, while a large dataset of 10,000 documents could take an hour or more. After training completes, Language Studio displays evaluation metrics. For classification, the metrics include precision, recall, and F1 score for each class and overall. For NER, the metrics also include precision, recall, and F1, computed at the entity level — a prediction is correct only if both the entity type and the exact text span match the ground truth.

Deploying and Consuming Custom Models via API

Once you are satisfied with the evaluation metrics, deploy the model in Language Studio. Deployment makes the model available at a prediction endpoint. You can deploy to a staging slot for testing and a production slot for live traffic. Each deployment has its own endpoint URL. This two-slot approach lets you validate a new model without disrupting production.

To consume the deployed model, call the prediction endpoint with your documents. The following Python example calls a custom text classification endpoint:

import os
from azure.core.credentials import AzureKeyCredential
from azure.ai.textanalytics import TextAnalyticsClient

endpoint = os.environ["LANGUAGE_ENDPOINT"]
key = os.environ["LANGUAGE_KEY"]
project_name = "support-ticket-classifier"
deployment_name = "production"

client = TextAnalyticsClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(key)
)

documents = [
    "My laptop screen flickers when connected to the charger.",
    "The invoice for September has incorrect tax calculations."
]

# Custom text classification uses the begin_* pattern
poller = client.begin_multi_label_classify(
    documents,
    project_name=project_name,
    deployment_name=deployment_name
)
results = list(poller.result())

for i, doc in enumerate(results):
    if not doc.is_error:
        print(f"Document {i+1} classifications:")
        for classification in doc.classifications:
            print(f"  {classification.category}: {classification.confidence_score:.2%}")
    else:
        print(f"Error: {doc.error}")

For custom NER, the API call is similar but uses the begin_recognize_custom_entities method:

poller = client.begin_recognize_custom_entities(
    documents,
    project_name=project_name,
    deployment_name=deployment_name
)
results = list(poller.result())

for i, doc in enumerate(results):
    if not doc.is_error:
        print(f"Document {i+1} entities:")
        for entity in doc.entities:
            print(f"  {entity.text} -> {entity.category} "
                  f"({entity.confidence_score:.2%})")
    else:
        print(f"Error: {doc.error}")

Iterative Improvement

Building a high-quality custom NLP model is an iterative process. You rarely get excellent results on the first training run. The typical improvement cycle involves analyzing prediction failures on the test set, identifying patterns in the errors, collecting additional labeled examples that address those patterns, retraining the model, and comparing the new evaluation metrics against the previous version.

Common improvement strategies include: adding more documents for classes with low recall (the model is not finding all instances), tightening label boundaries for entities with low precision (the model is including extra words in entity spans), removing ambiguous documents that contradict each other, adding negative examples (documents that belong to no class) for classification models, and checking for labeling inconsistencies where similar text is labeled differently across documents.

Language Studio's test set evaluation page provides a detailed breakdown of predictions versus ground truth. For each misclassified document, you can see what the model predicted and what the correct label was. This visibility helps you identify specific gaps in your training data.

Exporting and Sharing Custom Models

Custom models trained in Language Studio are tied to the Language resource in which they were created. You cannot directly export a model to another resource. However, you can export the training data and project configuration as a JSON file, then import it into another Language resource in a different region or subscription. The exported file contains the project schema, all labeled documents, and model configuration but not the trained model weights themselves. After importing, you must retrain the model in the new resource.

Patient Smith was prescribed Metformin 500mg twice daily for type 2 diabetes.
Dr. Johnson noted elevated blood pressure at 140/90 and started Lisinopril 10mg.
Chest X-ray showed no abnormalities. Patient reports occasional dizziness.

Summary

Custom text classification and custom NER extend Azure AI Language with domain-specific capabilities. Classification assigns single or multiple labels to documents, while NER extracts specific entity spans. Building a custom model requires creating a project in Language Studio, labeling data with your schema, training the model, evaluating the metrics, and deploying to a prediction endpoint. The API for consuming custom models uses an async polling pattern with project and deployment name parameters. Iterative improvement through error analysis and additional labeling is essential for production-quality results. Custom models cannot replace prebuilt APIs for general NLP tasks, but they dramatically outperform prebuilt models on domain-specific content. The next chapter covers Conversational Language Understanding for building intent and entity models for conversational applications.