AI-103 Text Analysis, Azure Translator and Speech Solutions

Introduction

Text analysis and speech on Azure combine structured extraction, translation and voice interaction. This lesson covers the whole Implement text analysis solutions domain of AI-103 (10–15% of the exam). The official skills fall into two groups. The first is language model text analysis: extracting entities, topics, summaries and structured JSON with generative prompting and Foundry Tools; detecting sentiment, tone and sensitive content; and translating text with Azure Translator or LLM-based flows, including domain customization. The second is speech solutions: speech-to-text and text-to-speech for agent interactions, custom speech models, speech translation, and audio-based multimodal reasoning.

After this lesson you will be able to choose between a large language model and a purpose-built language service for a text task, design a prompt that returns validated structured JSON, protect personal data, pick the right translation approach, and design a voice agent that listens, reasons and speaks with acceptable latency.

Learning objectives

• Extract entities, topics and summaries from text as schema-validated JSON.
• Detect sentiment, tone and sensitive content, and handle personal data correctly.
• Choose between Azure Translator, a customized translator and LLM translation.
• Apply domain customization (glossaries, custom models, prompt context).
• Design speech-to-text, text-to-speech and speech-translation components for an agent.
• Decide when custom speech is needed and when a multimodal audio model is a better fit.

1. Text analysis with language models

Classical text analytics (key phrases, named-entity recognition, sentiment) is available as dedicated Foundry Tools in the Azure Language service. Generative models add flexibility: you describe the task in natural language and supported structured-output deployments can constrain the response format, while extraction accuracy still needs evaluation. The exam expects you to know when each is the better fit.

1.1 Entities, topics and summaries

An entity is a named thing with a type: person, organization, product, date, amount. A topic is a theme of a text, often from a controlled list. A summary condenses the text for a purpose. With a model, each is a field in the output.

A robust prompt pattern:

1. State the role and the task ("You extract facts from support emails").
2. Give the allowed values for categorical fields (an enumerated list of topics).
3. Say what to do when information is absent ("use null, never guess").
4. Provide one or two short examples of input and output (few-shot).
5. Require the answer in a specific structure.

1.2 Structured JSON outputs

Free text is hard to use in code. Depending on the model, version and API, Foundry chat deployments may support JSON mode (valid JSON, no schema guarantee) and structured outputs (the response conforms to a supplied JSON schema). Use structured outputs whenever the model supports them for the deployment.

import os
from pydantic import BaseModel
from typing import Literal, Optional
from openai import OpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

client = OpenAI(
    base_url=os.environ["AZURE_OPENAI_BASE_URL"],  # resource URL ending /openai/v1/
    api_key=get_bearer_token_provider(DefaultAzureCredential(),
                                      "https://ai.azure.com/.default"),
)

class Ticket(BaseModel):
    topic: Literal["billing", "delivery", "product-defect", "other"]
    sentiment: Literal["positive", "neutral", "negative"]
    order_id: Optional[str]
    summary: str          # sentence count needs a separate application check
    contains_personal_data: bool

email_text = "Order AB1234 arrived damaged. Please help me arrange a replacement."
resp = client.chat.completions.parse(
    model=os.environ["AZURE_OPENAI_DEPLOYMENT"],
    messages=[
        {"role": "system", "content": "Extract the fields. Use null when the order id is absent. Never invent values."},
        {"role": "user", "content": email_text},
    ],
    response_format=Ticket,
)
choice = resp.choices[0]
if choice.finish_reason != "stop" or choice.message.refusal or choice.message.parsed is None:
    raise RuntimeError("Extraction incomplete or refused; route for review")
ticket = choice.message.parsed
print(ticket.model_dump_json())

Use a current compatible OpenAI Python SDK with Pydantic and azure-identity, a deployment supporting structured outputs, and an authenticated principal authorized for inference. Handle SDK parsing exceptions and API failures as reviewable failures, with bounded retries for transient errors. DefaultAzureCredential is a chain, not a guarantee of managed identity; choose the production identity explicitly.

Points to remember:

• The schema (with Literal enums) constrains the shape, not the truth. A model can still choose the wrong enum value, so evaluate accuracy on labelled samples.
• Where supported, a low temperature can reduce variability but does not guarantee deterministic extraction.
• Validate in code anyway (for example check that order_id matches your pattern) and route failures to a retry or a human.
• A refusal or a truncated response (finish reason length) can break parsing, so check the finish reason and give enough output tokens.

Worked example 1: triaging 20,000 support emails

Scenario. Contoso Telecom receives 20,000 emails a day in English and Spanish. It needs topic, sentiment, an order number when present, and a two-sentence English summary for agents. Emails sometimes contain card numbers. The budget is tight.

Reasoning.

1. Language detection and PII detection are fixed tasks at high volume, so use the Language service for both: detect the language and redact payment-card numbers before anything else.
2. Topic, summary and the order number benefit from flexible extraction, so evaluate a small chat deployment with structured outputs against task-specific alternatives.
3. The Spanish emails need an English summary; the model can summarise across languages in one call, so no separate translation step is needed.
4. Send only the redacted text to the model.
5. Check order-number syntax and existence in the business system, summary length and required evidence. Route failed checks or high-risk cases for review; a generated confidence number is not a calibrated probability.
6. Evaluate on 500 labelled emails: accuracy of topic, sentiment agreement and summary quality rated by agents.

Why the other choices are weaker. Sending raw emails to a general model exposes unnecessary card data and may add avoidable cost. A custom-trained classifier requires labelled data and retraining whenever topics change. Azure Language already supports abstractive as well as extractive summarization; the LLM choice here is for the combined custom schema and cross-language summary. Its summarization feature retires on 31 March 2029, so plan new summary workloads with the current Foundry guidance.

2. Sentiment, tone and sensitive content

2.1 Sentiment and tone

Sentiment is the polarity of a text (positive, neutral, negative), sometimes with aspect-level detail ("the food was great, the service was slow"). Tone is the manner of expression: polite, urgent, angry, sarcastic, formal. The Language service gives sentiment with confidence scores and opinion mining (aspects). A model can additionally return tone labels and explain them.

Good practice: define each label in the prompt, include borderline examples, keep the output enumerated, and measure inter-rater agreement between the model and human reviewers.

2.2 Sensitive content and personal data

"Sensitive" covers three different problems, and the exam likes to test that you separate them:

PII detection returns entity categories, offsets and redacted text according to the API configuration. It can miss entities; test languages and formats and combine it with deterministic checks where appropriate. Redact before logging, indexing or sending text to a model that does not need the data. Text redaction does not remove spoken personal data from the source recording. Define separate consent, access, audio-redaction and retention controls; avoiding persistence requires blocking recording or rejecting the sensitive segment before it is stored. Keep redaction rules versioned and test them on realistic samples, since a missed card number is a data-protection incident.

Important behaviours:

• Inference about protected characteristics (health, ethnicity, religion) from text is both unreliable and against responsible-AI guidance; do not build features that guess them.
• A content filter blocking a customer's complaint because it quotes abusive language is a false positive. Provide a review route and tune severity thresholds to your audience.

3. Translation

3.1 Two families of approach

Many production systems combine both: Translator for volume, an LLM pass for quality-critical text, and a human reviewer for legal content.

3.2 Domain customization

General translation fails on specialised vocabulary: "pitch" in aviation, "consideration" in contract law, product names that must not be translated. Options, from simplest to heaviest:

1. Phrase and glossary control. Supply a glossary of term pairs to document translation, or mark brand names so they stay untranslated (the dynamic dictionary in the Translator API).

2. Custom Translator training. A fully trained NMT model needs at least 10,000 unique aligned sentence pairs; deploy the trained model and use its category ID. Dictionary-only customization and adaptive custom translation are different workflows. Adaptive customization can use a much smaller reference dataset with supported models; check availability and evaluate it separately.

3. LLM instructions. Give the model the glossary, the audience and a style guide in the prompt, and ask for the same terms every time. Include few-shot examples.

4. Hybrid. Translate with Translator using a glossary, then ask a model to polish while keeping glossary terms fixed, or verify with a back-translation spot check.

For text translation, the dynamic dictionary is intended for compound nouns such as product names; the language pair must include English and the source language must be specified. It is not the same as a document-translation glossary. Check supported file types, language pairs and glossary behavior in the selected API.

Evaluate with a held-out set: automatic metrics (BLEU or COMET-style scores) for trends, and bilingual human review for quality. Metrics do not replace review for regulated content.

Worked example 2: translating a product manual

Scenario. A medical-device maker has 400 Word manuals to translate into five languages. Device names and regulatory phrases must remain exact. Layout and tables must be preserved. A reviewer will approve final text.

Reasoning.

1. Use document translation from Azure Translator so layout and formatting stay intact and Word files are processed in bulk from Blob Storage.
2. Provide a glossary with the approved regulatory phrases and the device names set to remain unchanged.
3. If glossaries are not enough after a pilot, train a custom translation model with approved earlier translations.
4. Add a reviewer workflow for final approval; keep source and target segments aligned for the next update.

Why not a bare LLM prompt for the whole manual? It does not provide the document API’s layout-preservation workflow and may paraphrase regulated terms; total cost depends on the pipeline. Verify translated layout and terminology rather than assuming perfect preservation. An LLM is still valuable for generating a plain-language summary or checking a sample for inconsistencies.

4. Speech solutions

Azure Speech (a Foundry Tool) provides four main capabilities: speech to text (STT, also called recognition or transcription), text to speech (TTS, synthesis), speech translation, and speaker-related features. Agents add a model in the middle.

4.1 The voice-agent loop

Microphone → STT (streaming) → agent / LLM (+ tools) → TTS (streaming) → Speaker

Latency is the central design issue: users notice delays above about a second or two. Techniques:

• Streaming recognition returns partial (hypothesis) results while the user speaks and a final result at the end of an utterance.
• Voice activity detection and barge-in: stop speaking when the user starts talking.
• Stream the model output into TTS sentence by sentence rather than waiting for the full answer.
• Keep answers short; use filler acknowledgements when a tool call is slow.
• Deploy resources in the region closest to users.

There are two architectural styles, and exam questions contrast them:

Voice Live is a managed voice-agent API, not synonymous with one end-to-end model: supported configurations expose controls such as phrase lists and custom speech. Benchmark the chosen configuration before deciding that only a cascade can meet audit or vocabulary requirements.

4.2 Speech to text

Three recognition modes matter:

Useful features: language identification (detect which language is spoken from a candidate list), diarization for call recordings, word-level timestamps, profanity filtering and phrase lists that bias recognition toward expected words (product names, acronyms) without any training. Phrase lists are supported by real-time and fast transcription and selected voice APIs, but not batch transcription; language/model support still applies.

4.3 Custom speech

Base models are trained on general speech. Custom speech adapts recognition to your domain. Choose the lightest option that fixes the problem:

Process: collect data, upload datasets, train, and compare word error rate (WER) on held-out audio. For real-time use, deploy a custom endpoint and reference its endpoint ID; batch transcription can use the custom model without a hosted endpoint. Training-data and pronunciation support vary by locale and base model. Compare WER of the base and custom models on audio that was not used in training before deploying.

4.4 Text to speech

Neural voices turn text into natural audio. Key concepts:

• SSML (Speech Synthesis Markup Language) controls voice, speaking style, rate, pitch, pauses and pronunciation (<phoneme>, <say-as> for dates, numbers, spelling).
• Streaming synthesis starts audio before the whole text is synthesised, which is what voice agents need.
• Custom neural voice builds a brand voice; it is a Limited Access capability that requires an approved application, recorded consent from the voice talent, and disclosure that the voice is synthetic.
• Visemes and word-boundary events are available for lip-sync avatars and caption highlighting.

<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US">
  <voice name="en-US-AvaMultilingualNeural">
    Your appointment is on <say-as interpret-as="date" format="mdy">10/21/2026</say-as>.
    <break time="300ms"/> Your reference is <say-as interpret-as="characters">AB12</say-as>.
  </voice>
</speak>

4.5 Speech translation

Speech translation recognises speech in one language and returns translated text in one or more target languages, and can synthesise the translation as audio. It suits live events, meetings and multilingual customer calls. Design points:

• Provide the source language, or use language identification across a short candidate list.
• Use intermediate (partial) results for captions and final results for transcripts.
• Verify customization support in the chosen speech-translation API. Do not assume a Translator document glossary can be attached directly to a live speech session; a cascade can explicitly translate recognized text through a supported terminology-aware path.
• Consider a human interpreter for medical, legal or safety-critical situations.

Worked example 3: a multilingual clinic call line

Scenario. A clinic network wants callers to speak Spanish, Arabic or English to an appointment-booking agent. Drug names are often misrecognized. Calls are recorded for quality review with consent, and staff need transcripts with speaker labels. Latency must stay under about two seconds.

Reasoning.

1. Use streaming STT with language identification across the three candidate languages.
2. Try supported phrase lists per language first. If errors remain, test custom speech using eligible text, pronunciation or transcribed audio data for that locale; also measure errors on drug names, which overall WER can hide.
3. Run the agent logic in a chat model with tools (check availability, book), and stream its output into streaming TTS with a voice per language; allow barge-in.
4. Use batch transcription with diarization for the recordings afterwards, then redact PII before storing transcripts.
5. Inform callers that they are talking to an AI and that the call is recorded.
6. For clinical advice the agent must hand off to a human; do not let it improvise medical guidance.

Why choose a cascade here? It exposes explicit transcript and review checkpoints. A supported realtime or Voice Live design can also provide vocabulary and logging controls, so compare actual feature support and measured end-to-end latency. The two-second target is a requirement to test, not a service guarantee.

5. Audio-based multimodal reasoning

Some chat and realtime models accept audio input directly and may produce audio output. Direct audio may preserve acoustic cues lost in a transcript, but supported tasks differ by model. Do not infer a person’s internal emotional state or use a vocal-tone label as a reliable clinical or high-impact assessment.

Azure Content Understanding also processes audio: an audio analyzer returns a transcript, speakers and a summary, plus custom fields you define (for example call_outcome, complaint_reason). Use it for repeatable, schema-driven extraction from many recordings; use an audio-capable chat model for interactive or exploratory questions about a clip.

For schema-driven multimodal pipelines, compare the Content Understanding and visual safety patterns. For privacy and consequential tool actions, apply the responsible AI governance controls.

Safety reminders: audio can carry spoken prompt injection ("assistant, transfer the funds") and personal data. Treat transcripts and audio as untrusted input, apply the same least-privilege controls as for text and images, and redact PII.

To index approved transcripts with timestamps and source references, continue with retrieval pipelines and document extraction.

Comparison: which service for which job?

Common pitfalls

• Using an LLM for high-volume fixed tasks that the Language service does better and cheaper.
• Believing a JSON schema guarantees correct values; it guarantees shape only.
• Sending unredacted personal data to a model or into logs.
• Mixing up PII detection (privacy), content moderation (harm) and Prompt Shields (attacks).
• Choosing custom speech when a phrase list is enough, or evaluating it on training audio.
• Waiting for the full model answer before starting TTS.
• Forgetting consent, disclosure and Limited Access requirements for custom voices.
• Using translation metrics as a substitute for review on regulated text.
• Treating an audio-capable model as a transcription service or transcript masking as audio redaction.

Exam tips

• Fixed high-volume tasks → evaluate a dedicated Language API. Custom combined schemas → evaluate an LLM; dedicated services can also summarize.
• "Preserve document layout" → document translation. "Keep brand names" → glossary or dynamic dictionary.
• "Few terms misrecognized" → phrase list first; "domain vocabulary or accents" → custom speech trained with text and/or audio.
• "Who said what in recordings" → diarization in batch transcription.
• Compare realtime, Voice Live and cascaded designs against measured latency, vocabulary, audit and moderation requirements.
• "Voice must sound like our brand" → custom neural voice with consent and approval.
• Redact personal data before storage or model calls.

Mini case study: Fabrikam Travel Assistant

Fabrikam Travel wants an assistant that (1) reads customer reviews in six languages and reports topic and sentiment weekly; (2) translates hotel descriptions into French and German with exact hotel names; (3) answers phone calls in English and Spanish about bookings; (4) transcribes agent calls with speaker labels for quality review; (5) never stores card numbers.

Task. Recommend components.

Solution.

1. Reviews: language detection from the Language service, then a chat model with structured outputs (topic enum, sentiment enum, aspect list) with supported generation settings. Evaluate against 400 human-labelled reviews per language group.
2. Hotel descriptions: Azure Translator with a glossary that keeps hotel names unchanged; an optional LLM polish pass for marketing tone with the glossary fixed; spot review by a bilingual editor.
3. Phone assistant: cascaded design: streaming STT with language identification (English/Spanish), phrase list for destination names, chat model with booking tools, streaming TTS with barge-in. Disclose that callers are talking to an AI.
4. Quality review: batch transcription with diarization only of recordings permitted by policy, then redaction of transcripts before ordinary storage. Transcript masking does not sanitize raw audio; disable or pause recording during payment and restrict any approved recording retention.
5. Card numbers: keep payment entry on a secure payment page and pause recording for payment handoff. Warn callers not to speak card data; use detection and rejection/redaction for accidental disclosures. PII detection alone cannot guarantee that no raw card data is ever stored.
6. Operations: managed identity, evaluation sets per language, cost tracking by pipeline, and alerts on content-safety detections.

Key terms

• Entity: a named item in text with a type, such as an organization or date.
• Structured outputs: model responses constrained to a JSON schema.
• PII: personally identifiable information.
• Opinion mining: aspect-level sentiment analysis.
• Glossary / dynamic dictionary: controls how specific terms are translated.
• Custom translation: a translation model trained on your parallel text.
• STT / TTS: speech to text and text to speech.
• Phrase list: runtime hints that bias recognition toward expected words.
• Custom speech: a recognition model adapted with your text and/or audio.
• WER: word error rate, the main recognition accuracy metric.
• Diarization: labelling which speaker spoke when.
• SSML: markup that controls synthesised speech.
• Barge-in: letting the user interrupt the agent's speech.
• Realtime model: a model that takes and produces audio directly with very low latency.

Summary checklist

• I can choose between the Language service and an LLM for a text task.
• I can extract entities, topics and summaries as schema-validated JSON and validate them in code.
• I can separate PII, harmful content and prompt-attack controls.
• I can choose between Translator, custom translation and LLM translation, and apply glossaries.
• I can design STT, LLM and TTS flows with streaming, barge-in and low latency.
• I can choose between phrase lists, custom speech and batch transcription with diarization.
• I know when an audio-capable model or a Content Understanding audio analyzer fits.

Sources

• Microsoft Learn, AI-103 study guide: https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ai-103 (opens in a new tab)
• Azure Language service documentation: https://learn.microsoft.com/en-us/azure/ai-services/language-service/ (opens in a new tab)
• Azure Translator documentation: https://learn.microsoft.com/en-us/azure/ai-services/translator/ (opens in a new tab)
• Azure Speech documentation: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/ (opens in a new tab)
• Structured outputs with Azure OpenAI in Foundry Models: https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/structured-outputs (opens in a new tab)
• Phrase-list support: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/improve-accuracy-phrase-list (opens in a new tab)
• Custom speech: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/custom-speech-overview (opens in a new tab)
• Custom voice and consent: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/custom-neural-voice (opens in a new tab)
• Voice Live capabilities: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live (opens in a new tab)
• Custom Translator training data: https://learn.microsoft.com/en-us/azure/ai-services/translator/custom-translator/concepts/parallel-documents (opens in a new tab)
• Adaptive custom translation: https://learn.microsoft.com/en-us/azure/ai-services/translator/foundry/adaptive-custom-translation (opens in a new tab)
• Dynamic dictionary restrictions: https://learn.microsoft.com/en-us/azure/ai-services/translator/text-translation/how-to/use-dynamic-dictionary (opens in a new tab)
• Summarization and lifecycle: https://learn.microsoft.com/en-us/azure/ai-services/language-service/summarization/overview (opens in a new tab)
• PII APIs: https://learn.microsoft.com/en-us/azure/ai-services/language-service/personally-identifiable-information/overview (opens in a new tab)
• Content Understanding audio overview: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/audio/overview (opens in a new tab)